fix(config): remove trailing newlines in regexes (#373)

Closes #372

Fun YAML fact of the day:

What is the difference between how these two expressions are parsed?

```yaml
foo: >
  bar
```

```yaml
foo: >-
  bar
```

They are invisible in yaml, but when you evaluate them to JSON the
difference is obvious:

```json
{
  "foo": "bar\n"
}
```

```json
{
  "foo": "bar"
}
```

User-Agent strings, URL path values, and HTTP headers _do_ end in
newlines in HTTP/1.1 wire form, but that newline is usually stripped
before the server actually handles it. Also HTTP/2 is a thing and does
not terminate header values with newlines.

This change makes Anubis more aggressively detect mistaken uses of the
yaml `>` operator and nudges the user into using the yaml `>-` operator
which does not append the trailing newline.

I had honestly forgotten about this YAML behavior because it wasn't
relevant for so long. Oops! Glad I released a beta.

Whenever you get into this state, Anubis will throw a config parsing
error and then give you a message hinting at the folly of your ways.

```
config.Bot: regular expression ends with newline (try >- instead of > in yaml)
```

Big thanks to https://yaml-multiline.info, this helped me realize my
folly instantly.

@aiverson, this is official permission to say "told you so".

Signed-off-by: Xe Iaso <me@xeiaso.net>

.github/workflows/docs-test.yml

@@ -0,0 +1,39 @@
+name: Docs test build
+
+on:
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+  actions: write
+
+jobs:
+  build:
+    runs-on: ubuntu-24.04
+
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
+        with:
+          images: ghcr.io/techarohq/anubis/docs
+
+      - name: Build and push
+        id: build
+        uses: docker/build-push-action@471d1dc4e07e5cdedd4c2171150001c434f0b7a4 # v6.15.0
+        with:
+          context: ./docs
+          cache-to: type=gha
+          cache-from: type=gha
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          platforms: linux/amd64
+          push: false

.github/workflows/package-builds-stable.yml

@@ -64,8 +64,9 @@ jobs:
 
     - name: Build Packages
       run: |
-        wget https://github.com/Xe/x/releases/download/v1.13.4/yeet_1.13.4_amd64.deb -O var/yeet.deb
+        wget https://github.com/TecharoHQ/yeet/releases/download/v0.1.1/yeet_0.1.1_amd64.deb -O var/yeet.deb
         sudo apt -y install -f ./var/yeet.deb
+        rm ./var/yeet.deb
         yeet
 
     - name: Upload released artifacts

.github/workflows/package-builds-unstable.yml

@@ -66,8 +66,9 @@ jobs:
 
     - name: Build Packages
       run: |
-        wget https://github.com/Xe/x/releases/download/v1.13.4/yeet_1.13.4_amd64.deb -O var/yeet.deb
+        wget https://github.com/TecharoHQ/yeet/releases/download/v0.1.1/yeet_0.1.1_amd64.deb -O var/yeet.deb
         sudo apt -y install -f ./var/yeet.deb
+        rm ./var/yeet.deb
         yeet
 
     - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2

.github/workflows/zizmor.yml

@@ -21,7 +21,7 @@ jobs:
           persist-credentials: false
 
       - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@0c5e2b8115b80b4c7c5ddf6ffdd634974642d182 # v5.4.1
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
 
       - name: Run zizmor 🌈
         run: uvx zizmor --format sarif . > results.sarif 

README.md

@@ -10,11 +10,19 @@
 ![language count](https://img.shields.io/github/languages/count/TecharoHQ/anubis)
 ![repo size](https://img.shields.io/github/repo-size/TecharoHQ/anubis)
 
-Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.
+## Sponsors
 
-Installing and using this will likely result in your website not being indexed by some search engines. This is considered a feature of Anubis, not a bug.
+Anubis is brought to you by sponsors and donors like:
 
-This is a bit of a nuclear response, but AI scraper bots scraping so aggressively have forced my hand. I hate that I have to do this, but this is what we get for the modern Internet because bots don't conform to standards like robots.txt, even when they claim to.
+[![Distrust](./docs/static/img/sponsors/distrust-logo.webp)](https://distrust.co)
+
+## Overview
+
+Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a proof-of-work challenge in order to protect upstream resources from scraper bots.
+
+This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
+
+Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](./admin/policies.mdx) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
 
 In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.
 
@@ -28,11 +36,17 @@ For live chat, please join the [Patreon](https://patreon.com/cadey) and ask in t
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date)](https://www.star-history.com/#TecharoHQ/anubis&Date)
+<a href="https://www.star-history.com/#TecharoHQ/anubis&Date">
+ <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date&theme=dark" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date" />
+   <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date" />
+ </picture>
+</a>
 
 ## Packaging Status
 
-[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg)](https://repology.org/project/anubis-anti-crawler/versions)
+[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg?columns=3)](https://repology.org/project/anubis-anti-crawler/versions)
 
 ## Contributors
 

anubis.go

@@ -1,4 +1,4 @@
-// Package Anubis contains the version number of Anubis.
+// Package anubis contains the version number of Anubis.
 package anubis
 
 // Version is the current version of Anubis.
@@ -11,9 +11,15 @@ var Version = "devel"
 // access.
 const CookieName = "within.website-x-cmd-anubis-auth"
 
+// BasePrefix is a global prefix for all Anubis endpoints. Can be emptied to remove the prefix entirely.
+var BasePrefix = ""
+
 // StaticPath is the location where all static Anubis assets are located.
 const StaticPath = "/.within.website/x/cmd/anubis/"
 
+// APIPrefix is the location where all Anubis API endpoints are located.
+const APIPrefix = "/.within.website/x/cmd/anubis/api/"
+
 // DefaultDifficulty is the default "difficulty" (number of leading zeroes)
 // that must be met by the client in order to pass the challenge.
 const DefaultDifficulty = 4

cmd/anubis/main.go

@@ -20,7 +20,6 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
-	"regexp"
 	"strconv"
 	"strings"
 	"sync"
@@ -28,6 +27,7 @@ import (
 	"time"
 
 	"github.com/TecharoHQ/anubis"
+	"github.com/TecharoHQ/anubis/data"
 	"github.com/TecharoHQ/anubis/internal"
 	libanubis "github.com/TecharoHQ/anubis/lib"
 	botPolicy "github.com/TecharoHQ/anubis/lib/policy"
@@ -38,6 +38,7 @@ import (
 )
 
 var (
+	basePrefix               = flag.String("base-prefix", "", "base prefix (root URL) the application is served under e.g. /myapp")
 	bind                     = flag.String("bind", ":8923", "network address to bind HTTP to")
 	bindNetwork              = flag.String("bind-network", "tcp", "network family to bind HTTP to, e.g. unix, tcp")
 	challengeDifficulty      = flag.Int("difficulty", anubis.DefaultDifficulty, "difficulty of the challenge")
@@ -50,15 +51,16 @@ var (
 	socketMode               = flag.String("socket-mode", "0770", "socket mode (permissions) for unix domain sockets.")
 	robotsTxt                = flag.Bool("serve-robots-txt", false, "serve a robots.txt file that disallows all robots")
 	policyFname              = flag.String("policy-fname", "", "full path to anubis policy document (defaults to a sensible built-in policy)")
+	redirectDomains          = flag.String("redirect-domains", "", "list of domains separated by commas which anubis is allowed to redirect to. Leaving this unset allows any domain.")
 	slogLevel                = flag.String("slog-level", "INFO", "logging level (see https://pkg.go.dev/log/slog#hdr-Levels)")
-	target                   = flag.String("target", "http://localhost:3923", "target to reverse proxy to")
+	target                   = flag.String("target", "http://localhost:3923", "target to reverse proxy to, set to an empty string to disable proxying when only using auth request")
 	healthcheck              = flag.Bool("healthcheck", false, "run a health check against Anubis")
 	useRemoteAddress         = flag.Bool("use-remote-address", false, "read the client's IP address from the network request, useful for debugging and running Anubis on bare metal")
 	debugBenchmarkJS         = flag.Bool("debug-benchmark-js", false, "respond to every request with a challenge for benchmarking hashrate")
 	ogPassthrough            = flag.Bool("og-passthrough", false, "enable Open Graph tag passthrough")
 	ogTimeToLive             = flag.Duration("og-expiry-time", 24*time.Hour, "Open Graph tag cache expiration time")
 	extractResources         = flag.String("extract-resources", "", "if set, extract the static resources to the specified folder")
-	webmasterEmail		 = flag.String("webmaster-email", "", "if set, displays webmaster's email on the reject page for appeals")
+	webmasterEmail           = flag.String("webmaster-email", "", "if set, displays webmaster's email on the reject page for appeals")
 )
 
 func keyFromHex(value string) (ed25519.PrivateKey, error) {
@@ -75,7 +77,7 @@ func keyFromHex(value string) (ed25519.PrivateKey, error) {
 }
 
 func doHealthCheck() error {
-	resp, err := http.Get("http://localhost" + *metricsBind + "/metrics")
+	resp, err := http.Get("http://localhost" + *metricsBind + anubis.BasePrefix + "/metrics")
 	if err != nil {
 		return fmt.Errorf("failed to fetch metrics: %w", err)
 	}
@@ -118,7 +120,10 @@ func setupListener(network string, address string) (net.Listener, string) {
 
 		err = os.Chmod(address, os.FileMode(mode))
 		if err != nil {
-			listener.Close()
+			err := listener.Close()
+			if err != nil {
+				log.Printf("failed to close listener: %v", err)
+			}
 			log.Fatal(fmt.Errorf("could not change socket mode: %w", err))
 		}
 	}
@@ -174,14 +179,10 @@ func main() {
 
 	internal.InitSlog(*slogLevel)
 
-	if *healthcheck {
-		if err := doHealthCheck(); err != nil {
+	if *extractResources != "" {
+		if err := extractEmbedFS(data.BotPolicies, ".", *extractResources); err != nil {
 			log.Fatal(err)
 		}
-		return
-	}
-
-	if *extractResources != "" {
 		if err := extractEmbedFS(web.Static, "static", *extractResources); err != nil {
 			log.Fatal(err)
 		}
@@ -189,9 +190,14 @@ func main() {
 		return
 	}
 
-	rp, err := makeReverseProxy(*target)
-	if err != nil {
-		log.Fatalf("can't make reverse proxy: %v", err)
+	var rp http.Handler
+	// when using anubis via Systemd and environment variables, then it is not possible to set targe to an empty string but only to space
+	if strings.TrimSpace(*target) != "" {
+		var err error
+		rp, err = makeReverseProxy(*target)
+		if err != nil {
+			log.Fatalf("can't make reverse proxy: %v", err)
+		}
 	}
 
 	policy, err := libanubis.LoadPoliciesOrDefault(*policyFname, *challengeDifficulty)
@@ -205,24 +211,24 @@ func main() {
 			continue
 		}
 
-		hash, err := rule.Hash()
-		if err != nil {
-			log.Fatalf("can't calculate checksum of rule %s: %v", rule.Name, err)
-		}
-
+		hash := rule.Hash()
 		fmt.Printf("* %s: %s\n", rule.Name, hash)
 	}
 	fmt.Println()
 
 	// replace the bot policy rules with a single rule that always benchmarks
 	if *debugBenchmarkJS {
-		userAgent := regexp.MustCompile(".")
 		policy.Bots = []botPolicy.Bot{{
-			Name:      "",
-			UserAgent: userAgent,
-			Action:    config.RuleBenchmark,
+			Name:   "",
+			Rules:  botPolicy.NewHeaderExistsChecker("User-Agent"),
+			Action: config.RuleBenchmark,
 		}}
 	}
+	if *basePrefix != "" && !strings.HasPrefix(*basePrefix, "/") {
+		log.Fatalf("[misconfiguration] base-prefix must start with a slash, eg: /%s", *basePrefix)
+	} else if strings.HasSuffix(*basePrefix, "/") {
+		log.Fatalf("[misconfiguration] base-prefix must not end with a slash")
+	}
 
 	var priv ed25519.PrivateKey
 	if *ed25519PrivateKeyHex != "" && *ed25519PrivateKeyHexFile != "" {
@@ -233,12 +239,12 @@ func main() {
 			log.Fatalf("failed to parse and validate ED25519_PRIVATE_KEY_HEX: %v", err)
 		}
 	} else if *ed25519PrivateKeyHexFile != "" {
-		hex, err := os.ReadFile(*ed25519PrivateKeyHexFile)
+		hexFile, err := os.ReadFile(*ed25519PrivateKeyHexFile)
 		if err != nil {
 			log.Fatalf("failed to read ED25519_PRIVATE_KEY_HEX_FILE %s: %v", *ed25519PrivateKeyHexFile, err)
 		}
 
-		priv, err = keyFromHex(string(bytes.TrimSpace(hex)))
+		priv, err = keyFromHex(string(bytes.TrimSpace(hexFile)))
 		if err != nil {
 			log.Fatalf("failed to parse and validate content of ED25519_PRIVATE_KEY_HEX_FILE: %v", err)
 		}
@@ -251,7 +257,22 @@ func main() {
 		slog.Warn("generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target, for more information: see https://anubis.techaro.lol/docs/admin/installation#key-generation")
 	}
 
+	var redirectDomainsList []string
+	if *redirectDomains != "" {
+		domains := strings.Split(*redirectDomains, ",")
+		for _, domain := range domains {
+			_, err = url.Parse(domain)
+			if err != nil {
+				log.Fatalf("cannot parse redirect-domain %q: %s", domain, err.Error())
+			}
+			redirectDomainsList = append(redirectDomainsList, strings.TrimSpace(domain))
+		}
+	} else {
+		slog.Warn("REDIRECT_DOMAINS is not set, Anubis will only redirect to the same domain a request is coming from, see https://anubis.techaro.lol/docs/admin/configuration/redirect-domains")
+	}
+
 	s, err := libanubis.New(libanubis.Options{
+		BasePrefix:        *basePrefix,
 		Next:              rp,
 		Policy:            policy,
 		ServeRobotsTXT:    *robotsTxt,
@@ -260,8 +281,9 @@ func main() {
 		CookiePartitioned: *cookiePartitioned,
 		OGPassthrough:     *ogPassthrough,
 		OGTimeToLive:      *ogTimeToLive,
+		RedirectDomains:   redirectDomainsList,
 		Target:            *target,
-		WebmasterEmail:     *webmasterEmail,
+		WebmasterEmail:    *webmasterEmail,
 	})
 	if err != nil {
 		log.Fatalf("can't construct libanubis.Server: %v", err)
@@ -276,13 +298,13 @@ func main() {
 		wg.Add(1)
 		go metricsServer(ctx, wg.Done)
 	}
-
 	go startDecayMapCleanup(ctx, s)
 
 	var h http.Handler
 	h = s
 	h = internal.RemoteXRealIP(*useRemoteAddress, *bindNetwork, h)
 	h = internal.XForwardedForToXRealIP(h)
+	h = internal.XForwardedForUpdate(h)
 
 	srv := http.Server{Handler: h}
 	listener, listenerUrl := setupListener(*bindNetwork, *bind)
@@ -297,6 +319,7 @@ func main() {
 		"debug-benchmark-js", *debugBenchmarkJS,
 		"og-passthrough", *ogPassthrough,
 		"og-expiry-time", *ogTimeToLive,
+		"base-prefix", *basePrefix,
 	)
 
 	go func() {
@@ -318,12 +341,20 @@ func metricsServer(ctx context.Context, done func()) {
 	defer done()
 
 	mux := http.NewServeMux()
-	mux.Handle("/metrics", promhttp.Handler())
+	mux.Handle(anubis.BasePrefix+"/metrics", promhttp.Handler())
 
 	srv := http.Server{Handler: mux}
 	listener, metricsUrl := setupListener(*metricsBindNetwork, *metricsBind)
 	slog.Debug("listening for metrics", "url", metricsUrl)
 
+	if *healthcheck {
+		log.Println("running healthcheck")
+		if err := doHealthCheck(); err != nil {
+			log.Fatal(err)
+		}
+		return
+	}
+
 	go func() {
 		<-ctx.Done()
 		c, cancel := context.WithTimeout(context.Background(), 5*time.Second)
@@ -349,7 +380,7 @@ func extractEmbedFS(fsys embed.FS, root string, destDir string) error {
 			return err
 		}
 
-		destPath := filepath.Join(destDir, relPath)
+		destPath := filepath.Join(destDir, root, relPath)
 
 		if d.IsDir() {
 			return os.MkdirAll(destPath, 0o700)

cmd/containerbuild/main.go

@@ -131,7 +131,7 @@ func parseImageList(imageList string) ([]image, error) {
 	}
 
 	if len(result) == 0 {
-		return nil, fmt.Errorf("no images provided, bad flags??")
+		return nil, fmt.Errorf("no images provided, bad flags")
 	}
 
 	return result, nil

data/apps/gitea-rss-feeds.yaml

@@ -0,0 +1,7 @@
+# By Aibrew: https://github.com/TecharoHQ/anubis/discussions/261#discussioncomment-12821065
+- name: gitea-feed-atom
+  action: ALLOW
+  path_regex: ^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*\.atom$
+- name: gitea-feed-rss
+  action: ALLOW
+  path_regex: ^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*\.rss$

data/botPolicies.json

@@ -1,677 +1,43 @@
 {
   "bots": [
     {
-      "name": "ai-robots-txt",
-      "user_agent_regex": "AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo Bot|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|PanguBot|Perplexity-User|PerplexityBot|PetalBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YouBot",
-      "action": "DENY"
+      "import": "(data)/bots/ai-robots-txt.yaml"
     },
     {
-      "name": "googlebot",
-      "user_agent_regex": "\\+http\\://www\\.google\\.com/bot\\.html",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "2001:4860:4801:10::/64",
-        "2001:4860:4801:11::/64",
-        "2001:4860:4801:12::/64",
-        "2001:4860:4801:13::/64",
-        "2001:4860:4801:14::/64",
-        "2001:4860:4801:15::/64",
-        "2001:4860:4801:16::/64",
-        "2001:4860:4801:17::/64",
-        "2001:4860:4801:18::/64",
-        "2001:4860:4801:19::/64",
-        "2001:4860:4801:1a::/64",
-        "2001:4860:4801:1b::/64",
-        "2001:4860:4801:1c::/64",
-        "2001:4860:4801:1d::/64",
-        "2001:4860:4801:1e::/64",
-        "2001:4860:4801:1f::/64",
-        "2001:4860:4801:20::/64",
-        "2001:4860:4801:21::/64",
-        "2001:4860:4801:22::/64",
-        "2001:4860:4801:23::/64",
-        "2001:4860:4801:24::/64",
-        "2001:4860:4801:25::/64",
-        "2001:4860:4801:26::/64",
-        "2001:4860:4801:27::/64",
-        "2001:4860:4801:28::/64",
-        "2001:4860:4801:29::/64",
-        "2001:4860:4801:2::/64",
-        "2001:4860:4801:2a::/64",
-        "2001:4860:4801:2b::/64",
-        "2001:4860:4801:2c::/64",
-        "2001:4860:4801:2d::/64",
-        "2001:4860:4801:2e::/64",
-        "2001:4860:4801:2f::/64",
-        "2001:4860:4801:31::/64",
-        "2001:4860:4801:32::/64",
-        "2001:4860:4801:33::/64",
-        "2001:4860:4801:34::/64",
-        "2001:4860:4801:35::/64",
-        "2001:4860:4801:36::/64",
-        "2001:4860:4801:37::/64",
-        "2001:4860:4801:38::/64",
-        "2001:4860:4801:39::/64",
-        "2001:4860:4801:3a::/64",
-        "2001:4860:4801:3b::/64",
-        "2001:4860:4801:3c::/64",
-        "2001:4860:4801:3d::/64",
-        "2001:4860:4801:3e::/64",
-        "2001:4860:4801:40::/64",
-        "2001:4860:4801:41::/64",
-        "2001:4860:4801:42::/64",
-        "2001:4860:4801:43::/64",
-        "2001:4860:4801:44::/64",
-        "2001:4860:4801:45::/64",
-        "2001:4860:4801:46::/64",
-        "2001:4860:4801:47::/64",
-        "2001:4860:4801:48::/64",
-        "2001:4860:4801:49::/64",
-        "2001:4860:4801:4a::/64",
-        "2001:4860:4801:4b::/64",
-        "2001:4860:4801:4c::/64",
-        "2001:4860:4801:50::/64",
-        "2001:4860:4801:51::/64",
-        "2001:4860:4801:52::/64",
-        "2001:4860:4801:53::/64",
-        "2001:4860:4801:54::/64",
-        "2001:4860:4801:55::/64",
-        "2001:4860:4801:56::/64",
-        "2001:4860:4801:60::/64",
-        "2001:4860:4801:61::/64",
-        "2001:4860:4801:62::/64",
-        "2001:4860:4801:63::/64",
-        "2001:4860:4801:64::/64",
-        "2001:4860:4801:65::/64",
-        "2001:4860:4801:66::/64",
-        "2001:4860:4801:67::/64",
-        "2001:4860:4801:68::/64",
-        "2001:4860:4801:69::/64",
-        "2001:4860:4801:6a::/64",
-        "2001:4860:4801:6b::/64",
-        "2001:4860:4801:6c::/64",
-        "2001:4860:4801:6d::/64",
-        "2001:4860:4801:6e::/64",
-        "2001:4860:4801:6f::/64",
-        "2001:4860:4801:70::/64",
-        "2001:4860:4801:71::/64",
-        "2001:4860:4801:72::/64",
-        "2001:4860:4801:73::/64",
-        "2001:4860:4801:74::/64",
-        "2001:4860:4801:75::/64",
-        "2001:4860:4801:76::/64",
-        "2001:4860:4801:77::/64",
-        "2001:4860:4801:78::/64",
-        "2001:4860:4801:79::/64",
-        "2001:4860:4801:80::/64",
-        "2001:4860:4801:81::/64",
-        "2001:4860:4801:82::/64",
-        "2001:4860:4801:83::/64",
-        "2001:4860:4801:84::/64",
-        "2001:4860:4801:85::/64",
-        "2001:4860:4801:86::/64",
-        "2001:4860:4801:87::/64",
-        "2001:4860:4801:88::/64",
-        "2001:4860:4801:90::/64",
-        "2001:4860:4801:91::/64",
-        "2001:4860:4801:92::/64",
-        "2001:4860:4801:93::/64",
-        "2001:4860:4801:94::/64",
-        "2001:4860:4801:95::/64",
-        "2001:4860:4801:96::/64",
-        "2001:4860:4801:a0::/64",
-        "2001:4860:4801:a1::/64",
-        "2001:4860:4801:a2::/64",
-        "2001:4860:4801:a3::/64",
-        "2001:4860:4801:a4::/64",
-        "2001:4860:4801:a5::/64",
-        "2001:4860:4801:c::/64",
-        "2001:4860:4801:f::/64",
-        "192.178.5.0/27",
-        "192.178.6.0/27",
-        "192.178.6.128/27",
-        "192.178.6.160/27",
-        "192.178.6.192/27",
-        "192.178.6.32/27",
-        "192.178.6.64/27",
-        "192.178.6.96/27",
-        "34.100.182.96/28",
-        "34.101.50.144/28",
-        "34.118.254.0/28",
-        "34.118.66.0/28",
-        "34.126.178.96/28",
-        "34.146.150.144/28",
-        "34.147.110.144/28",
-        "34.151.74.144/28",
-        "34.152.50.64/28",
-        "34.154.114.144/28",
-        "34.155.98.32/28",
-        "34.165.18.176/28",
-        "34.175.160.64/28",
-        "34.176.130.16/28",
-        "34.22.85.0/27",
-        "34.64.82.64/28",
-        "34.65.242.112/28",
-        "34.80.50.80/28",
-        "34.88.194.0/28",
-        "34.89.10.80/28",
-        "34.89.198.80/28",
-        "34.96.162.48/28",
-        "35.247.243.240/28",
-        "66.249.64.0/27",
-        "66.249.64.128/27",
-        "66.249.64.160/27",
-        "66.249.64.224/27",
-        "66.249.64.32/27",
-        "66.249.64.64/27",
-        "66.249.64.96/27",
-        "66.249.65.0/27",
-        "66.249.65.128/27",
-        "66.249.65.160/27",
-        "66.249.65.192/27",
-        "66.249.65.224/27",
-        "66.249.65.32/27",
-        "66.249.65.64/27",
-        "66.249.65.96/27",
-        "66.249.66.0/27",
-        "66.249.66.128/27",
-        "66.249.66.160/27",
-        "66.249.66.192/27",
-        "66.249.66.224/27",
-        "66.249.66.32/27",
-        "66.249.66.64/27",
-        "66.249.66.96/27",
-        "66.249.68.0/27",
-        "66.249.68.128/27",
-        "66.249.68.32/27",
-        "66.249.68.64/27",
-        "66.249.68.96/27",
-        "66.249.69.0/27",
-        "66.249.69.128/27",
-        "66.249.69.160/27",
-        "66.249.69.192/27",
-        "66.249.69.224/27",
-        "66.249.69.32/27",
-        "66.249.69.64/27",
-        "66.249.69.96/27",
-        "66.249.70.0/27",
-        "66.249.70.128/27",
-        "66.249.70.160/27",
-        "66.249.70.192/27",
-        "66.249.70.224/27",
-        "66.249.70.32/27",
-        "66.249.70.64/27",
-        "66.249.70.96/27",
-        "66.249.71.0/27",
-        "66.249.71.128/27",
-        "66.249.71.160/27",
-        "66.249.71.192/27",
-        "66.249.71.224/27",
-        "66.249.71.32/27",
-        "66.249.71.64/27",
-        "66.249.71.96/27",
-        "66.249.72.0/27",
-        "66.249.72.128/27",
-        "66.249.72.160/27",
-        "66.249.72.192/27",
-        "66.249.72.224/27",
-        "66.249.72.32/27",
-        "66.249.72.64/27",
-        "66.249.72.96/27",
-        "66.249.73.0/27",
-        "66.249.73.128/27",
-        "66.249.73.160/27",
-        "66.249.73.192/27",
-        "66.249.73.224/27",
-        "66.249.73.32/27",
-        "66.249.73.64/27",
-        "66.249.73.96/27",
-        "66.249.74.0/27",
-        "66.249.74.128/27",
-        "66.249.74.160/27",
-        "66.249.74.192/27",
-        "66.249.74.32/27",
-        "66.249.74.64/27",
-        "66.249.74.96/27",
-        "66.249.75.0/27",
-        "66.249.75.128/27",
-        "66.249.75.160/27",
-        "66.249.75.192/27",
-        "66.249.75.224/27",
-        "66.249.75.32/27",
-        "66.249.75.64/27",
-        "66.249.75.96/27",
-        "66.249.76.0/27",
-        "66.249.76.128/27",
-        "66.249.76.160/27",
-        "66.249.76.192/27",
-        "66.249.76.224/27",
-        "66.249.76.32/27",
-        "66.249.76.64/27",
-        "66.249.76.96/27",
-        "66.249.77.0/27",
-        "66.249.77.128/27",
-        "66.249.77.160/27",
-        "66.249.77.192/27",
-        "66.249.77.224/27",
-        "66.249.77.32/27",
-        "66.249.77.64/27",
-        "66.249.77.96/27",
-        "66.249.78.0/27",
-        "66.249.78.32/27",
-        "66.249.79.0/27",
-        "66.249.79.128/27",
-        "66.249.79.160/27",
-        "66.249.79.192/27",
-        "66.249.79.224/27",
-        "66.249.79.32/27",
-        "66.249.79.64/27",
-        "66.249.79.96/27"
-      ]
+      "import": "(data)/bots/cloudflare-workers.yaml"
     },
     {
-      "name": "bingbot",
-      "user_agent_regex": "\\+http\\://www\\.bing\\.com/bingbot\\.htm",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "157.55.39.0/24",
-        "207.46.13.0/24",
-        "40.77.167.0/24",
-        "13.66.139.0/24",
-        "13.66.144.0/24",
-        "52.167.144.0/24",
-        "13.67.10.16/28",
-        "13.69.66.240/28",
-        "13.71.172.224/28",
-        "139.217.52.0/28",
-        "191.233.204.224/28",
-        "20.36.108.32/28",
-        "20.43.120.16/28",
-        "40.79.131.208/28",
-        "40.79.186.176/28",
-        "52.231.148.0/28",
-        "20.79.107.240/28",
-        "51.105.67.0/28",
-        "20.125.163.80/28",
-        "40.77.188.0/22",
-        "65.55.210.0/24",
-        "199.30.24.0/23",
-        "40.77.202.0/24",
-        "40.77.139.0/25",
-        "20.74.197.0/28",
-        "20.15.133.160/27",
-        "40.77.177.0/24",
-        "40.77.178.0/23"
-      ]
+      "import": "(data)/bots/headless-browsers.yaml"
     },
     {
-      "name": "duckduckbot",
-      "user_agent_regex": "\\+http\\://duckduckgo\\.com/duckduckbot\\.html",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "57.152.72.128/32",
-        "51.8.253.152/32",
-        "40.80.242.63/32",
-        "20.12.141.99/32",
-        "20.49.136.28/32",
-        "51.116.131.221/32",
-        "51.107.40.209/32",
-        "20.40.133.240/32",
-        "20.50.168.91/32",
-        "51.120.48.122/32",
-        "20.193.45.113/32",
-        "40.76.173.151/32",
-        "40.76.163.7/32",
-        "20.185.79.47/32",
-        "52.142.26.175/32",
-        "20.185.79.15/32",
-        "52.142.24.149/32",
-        "40.76.162.208/32",
-        "40.76.163.23/32",
-        "40.76.162.191/32",
-        "40.76.162.247/32",
-        "40.88.21.235/32",
-        "20.191.45.212/32",
-        "52.146.59.12/32",
-        "52.146.59.156/32",
-        "52.146.59.154/32",
-        "52.146.58.236/32",
-        "20.62.224.44/32",
-        "51.104.180.53/32",
-        "51.104.180.47/32",
-        "51.104.180.26/32",
-        "51.104.146.225/32",
-        "51.104.146.235/32",
-        "20.73.202.147/32",
-        "20.73.132.240/32",
-        "20.71.12.143/32",
-        "20.56.197.58/32",
-        "20.56.197.63/32",
-        "20.43.150.93/32",
-        "20.43.150.85/32",
-        "20.44.222.1/32",
-        "40.89.243.175/32",
-        "13.89.106.77/32",
-        "52.143.242.6/32",
-        "52.143.241.111/32",
-        "52.154.60.82/32",
-        "20.197.209.11/32",
-        "20.197.209.27/32",
-        "20.226.133.105/32",
-        "191.234.216.4/32",
-        "191.234.216.178/32",
-        "20.53.92.211/32",
-        "20.53.91.2/32",
-        "20.207.99.197/32",
-        "20.207.97.190/32",
-        "40.81.250.205/32",
-        "40.64.106.11/32",
-        "40.64.105.247/32",
-        "20.72.242.93/32",
-        "20.99.255.235/32",
-        "20.113.3.121/32",
-        "52.224.16.221/32",
-        "52.224.21.53/32",
-        "52.224.20.204/32",
-        "52.224.21.19/32",
-        "52.224.20.249/32",
-        "52.224.20.203/32",
-        "52.224.20.190/32",
-        "52.224.16.229/32",
-        "52.224.21.20/32",
-        "52.146.63.80/32",
-        "52.224.20.227/32",
-        "52.224.20.193/32",
-        "52.190.37.160/32",
-        "52.224.21.23/32",
-        "52.224.20.223/32",
-        "52.224.20.181/32",
-        "52.224.21.49/32",
-        "52.224.21.55/32",
-        "52.224.21.61/32",
-        "52.224.19.152/32",
-        "52.224.20.186/32",
-        "52.224.21.27/32",
-        "52.224.21.51/32",
-        "52.224.20.174/32",
-        "52.224.21.4/32",
-        "51.104.164.109/32",
-        "51.104.167.71/32",
-        "51.104.160.177/32",
-        "51.104.162.149/32",
-        "51.104.167.95/32",
-        "51.104.167.54/32",
-        "51.104.166.111/32",
-        "51.104.167.88/32",
-        "51.104.161.32/32",
-        "51.104.163.250/32",
-        "51.104.164.189/32",
-        "51.104.167.19/32",
-        "51.104.160.167/32",
-        "51.104.167.110/32",
-        "20.191.44.119/32",
-        "51.104.167.104/32",
-        "20.191.44.234/32",
-        "51.104.164.215/32",
-        "51.104.167.52/32",
-        "20.191.44.22/32",
-        "51.104.167.87/32",
-        "51.104.167.96/32",
-        "20.191.44.16/32",
-        "51.104.167.61/32",
-        "51.104.164.147/32",
-        "20.50.48.159/32",
-        "40.114.182.172/32",
-        "20.50.50.130/32",
-        "20.50.50.163/32",
-        "20.50.50.46/32",
-        "40.114.182.153/32",
-        "20.50.50.118/32",
-        "20.50.49.55/32",
-        "20.50.49.25/32",
-        "40.114.183.251/32",
-        "20.50.50.123/32",
-        "20.50.49.237/32",
-        "20.50.48.192/32",
-        "20.50.50.134/32",
-        "51.138.90.233/32",
-        "40.114.183.196/32",
-        "20.50.50.146/32",
-        "40.114.183.88/32",
-        "20.50.50.145/32",
-        "20.50.50.121/32",
-        "20.50.49.40/32",
-        "51.138.90.206/32",
-        "40.114.182.45/32",
-        "51.138.90.161/32",
-        "20.50.49.0/32",
-        "40.119.232.215/32",
-        "104.43.55.167/32",
-        "40.119.232.251/32",
-        "40.119.232.50/32",
-        "40.119.232.146/32",
-        "40.119.232.218/32",
-        "104.43.54.127/32",
-        "104.43.55.117/32",
-        "104.43.55.116/32",
-        "104.43.55.166/32",
-        "52.154.169.50/32",
-        "52.154.171.70/32",
-        "52.154.170.229/32",
-        "52.154.170.113/32",
-        "52.154.171.44/32",
-        "52.154.172.2/32",
-        "52.143.244.81/32",
-        "52.154.171.87/32",
-        "52.154.171.250/32",
-        "52.154.170.28/32",
-        "52.154.170.122/32",
-        "52.143.243.117/32",
-        "52.143.247.235/32",
-        "52.154.171.235/32",
-        "52.154.171.196/32",
-        "52.154.171.0/32",
-        "52.154.170.243/32",
-        "52.154.170.26/32",
-        "52.154.169.200/32",
-        "52.154.170.96/32",
-        "52.154.170.88/32",
-        "52.154.171.150/32",
-        "52.154.171.205/32",
-        "52.154.170.117/32",
-        "52.154.170.209/32",
-        "191.235.202.48/32",
-        "191.233.3.202/32",
-        "191.235.201.214/32",
-        "191.233.3.197/32",
-        "191.235.202.38/32",
-        "20.53.78.144/32",
-        "20.193.24.10/32",
-        "20.53.78.236/32",
-        "20.53.78.138/32",
-        "20.53.78.123/32",
-        "20.53.78.106/32",
-        "20.193.27.215/32",
-        "20.193.25.197/32",
-        "20.193.12.126/32",
-        "20.193.24.251/32",
-        "20.204.242.101/32",
-        "20.207.72.113/32",
-        "20.204.242.19/32",
-        "20.219.45.67/32",
-        "20.207.72.11/32",
-        "20.219.45.190/32",
-        "20.204.243.55/32",
-        "20.204.241.148/32",
-        "20.207.72.110/32",
-        "20.204.240.172/32",
-        "20.207.72.21/32",
-        "20.204.246.81/32",
-        "20.207.107.181/32",
-        "20.204.246.254/32",
-        "20.219.43.246/32",
-        "52.149.25.43/32",
-        "52.149.61.51/32",
-        "52.149.58.139/32",
-        "52.149.60.38/32",
-        "52.148.165.38/32",
-        "52.143.95.162/32",
-        "52.149.56.151/32",
-        "52.149.30.45/32",
-        "52.149.58.173/32",
-        "52.143.95.204/32",
-        "52.149.28.83/32",
-        "52.149.58.69/32",
-        "52.148.161.87/32",
-        "52.149.58.27/32",
-        "52.149.28.18/32",
-        "20.79.226.26/32",
-        "20.79.239.66/32",
-        "20.79.238.198/32",
-        "20.113.14.159/32",
-        "20.75.144.152/32",
-        "20.43.172.120/32",
-        "20.53.134.160/32",
-        "20.201.15.208/32",
-        "20.93.28.24/32",
-        "20.61.34.40/32",
-        "52.242.224.168/32",
-        "20.80.129.80/32",
-        "20.195.108.47/32",
-        "4.195.133.120/32",
-        "4.228.76.163/32",
-        "4.182.131.108/32",
-        "4.209.224.56/32",
-        "108.141.83.74/32",
-        "4.213.46.14/32",
-        "172.169.17.165/32",
-        "51.8.71.117/32",
-        "20.3.1.178/32",
-        "52.149.56.151/32",
-        "52.149.30.45/32",
-        "52.149.58.173/32",
-        "52.143.95.204/32",
-        "52.149.28.83/32",
-        "52.149.58.69/32",
-        "52.148.161.87/32",
-        "52.149.58.27/32",
-        "52.149.28.18/32",
-        "20.79.226.26/32",
-        "20.79.239.66/32",
-        "20.79.238.198/32",
-        "20.113.14.159/32",
-        "20.75.144.152/32",
-        "20.43.172.120/32",
-        "20.53.134.160/32",
-        "20.201.15.208/32",
-        "20.93.28.24/32",
-        "20.61.34.40/32",
-        "52.242.224.168/32",
-        "20.80.129.80/32",
-        "20.195.108.47/32",
-        "4.195.133.120/32",
-        "4.228.76.163/32",
-        "4.182.131.108/32",
-        "4.209.224.56/32",
-        "108.141.83.74/32",
-        "4.213.46.14/32",
-        "172.169.17.165/32",
-        "51.8.71.117/32",
-        "20.3.1.178/32"
-      ]
+      "import": "(data)/bots/us-ai-scraper.yaml"
     },
     {
-      "name": "qwantbot",
-      "user_agent_regex": "\\+https\\://help\\.qwant\\.com/bot/",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "91.242.162.0/24"
-      ]
+      "import": "(data)/crawlers/googlebot.yaml"
     },
     {
-      "name": "internet-archive",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "207.241.224.0/20",
-        "208.70.24.0/21",
-        "2620:0:9c0::/48"
-      ]
+      "import": "(data)/crawlers/bingbot.yaml"
     },
     {
-      "name": "kagibot",
-      "user_agent_regex": "\\+https\\://kagi\\.com/bot",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "216.18.205.234/32",
-        "35.212.27.76/32",
-        "104.254.65.50/32",
-        "209.151.156.194/32"
-      ]
+      "import": "(data)/crawlers/duckduckbot.yaml"
     },
     {
-      "name": "marginalia",
-      "user_agent_regex": "search\\.marginalia\\.nu",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "193.183.0.162/31",
-        "193.183.0.164/30",
-        "193.183.0.168/30",
-        "193.183.0.172/31",
-        "193.183.0.174/32"
-      ]
+      "import": "(data)/crawlers/qwantbot.yaml"
     },
     {
-      "name": "mojeekbot",
-      "user_agent_regex": "http\\://www\\.mojeek\\.com/bot\\.html",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "5.102.173.71/32"
-      ]
+      "import": "(data)/crawlers/internet-archive.yaml"
     },
     {
-      "name": "us-artificial-intelligence-scraper",
-      "user_agent_regex": "\\+https\\://github\\.com/US-Artificial-Intelligence/scraper",
-      "action": "DENY"
+      "import": "(data)/crawlers/kagibot.yaml"
     },
     {
-      "name": "well-known",
-      "path_regex": "^/.well-known/.*$",
-      "action": "ALLOW"
+      "import": "(data)/crawlers/marginalia.yaml"
     },
     {
-      "name": "favicon",
-      "path_regex": "^/favicon.ico$",
-      "action": "ALLOW"
+      "import": "(data)/crawlers/mojeekbot.yaml"
     },
     {
-      "name": "robots-txt",
-      "path_regex": "^/robots.txt$",
-      "action": "ALLOW"
-    },
-    {
-      "name": "lightpanda",
-      "user_agent_regex": "^Lightpanda/.*$",
-      "action": "DENY"
-    },
-    {
-      "name": "headless-chrome",
-      "user_agent_regex": "HeadlessChrome",
-      "action": "DENY"
-    },
-    {
-      "name": "headless-chromium",
-      "user_agent_regex": "HeadlessChromium",
-      "action": "DENY"
-    },
-    {
-      "name": "generic-bot-catchall",
-      "user_agent_regex": "(?i:bot|crawler)",
-      "action": "CHALLENGE",
-      "challenge": {
-        "difficulty": 16,
-        "report_as": 4,
-        "algorithm": "slow"
-      }
+      "import": "(data)/common/keep-internet-working.yaml"
     },
     {
       "name": "generic-browser",
@@ -680,4 +46,4 @@
     }
   ],
   "dnsbl": false
-}
+}

data/botPolicies.yaml

@@ -0,0 +1,50 @@
+## Anubis has the ability to let you import snippets of configuration into the main
+## configuration file. This allows you to break up your config into smaller parts
+## that get logically assembled into one big file.
+##
+## Of note, a bot rule can either have inline bot configuration or import a
+## bot config snippet. You cannot do both in a single bot rule.
+##
+## Import paths can either be prefixed with (data) to import from the common/shared
+## rules in the data folder in the Anubis source tree or will point to absolute/relative
+## paths in your filesystem. If you don't have access to the Anubis source tree, check
+## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
+
+bots:
+# Pathological bots to deny
+- # This correlates to data/bots/ai-robots-txt.yaml in the source tree
+  import: (data)/bots/ai-robots-txt.yaml
+- import: (data)/bots/cloudflare-workers.yaml 
+- import: (data)/bots/headless-browsers.yaml
+- import: (data)/bots/us-ai-scraper.yaml
+
+# Search engines to allow
+- import: (data)/crawlers/googlebot.yaml
+- import: (data)/crawlers/bingbot.yaml
+- import: (data)/crawlers/duckduckbot.yaml
+- import: (data)/crawlers/qwantbot.yaml
+- import: (data)/crawlers/internet-archive.yaml
+- import: (data)/crawlers/kagibot.yaml
+- import: (data)/crawlers/marginalia.yaml
+- import: (data)/crawlers/mojeekbot.yaml
+
+# Allow common "keeping the internet working" routes (well-known, favicon, robots.txt)
+- import: (data)/common/keep-internet-working.yaml
+
+# # Punish any bot with "bot" in the user-agent string
+# # This is known to have a high false-positive rate, use at your own risk
+# - name: generic-bot-catchall
+#   user_agent_regex: (?i:bot|crawler)
+#   action: CHALLENGE
+#   challenge:
+#     difficulty: 16  # impossible
+#     report_as: 4    # lie to the operator
+#     algorithm: slow # intentionally waste CPU cycles and time
+
+# Generic catchall rule
+- name: generic-browser
+  user_agent_regex: >-
+    Mozilla|Opera
+  action: CHALLENGE
+
+dnsbl: false

data/bots/ai-robots-txt.yaml

@@ -0,0 +1,4 @@
+- name: "ai-robots-txt"
+  user_agent_regex: >-
+    AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo Bot|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|PanguBot|Perplexity-User|PerplexityBot|PetalBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YouBot
+  action: DENY

data/bots/cloudflare-workers.yaml

@@ -0,0 +1,4 @@
+- name: cloudflare-workers
+  headers_regex:
+    CF-Worker: .*
+  action: DENY

data/bots/headless-browsers.yaml

@@ -0,0 +1,9 @@
+- name: lightpanda
+  user_agent_regex: ^LightPanda/.*$
+  action: DENY
+- name: headless-chrome
+  user_agent_regex: HeadlessChrome
+  action: DENY
+- name: headless-chromium
+  user_agent_regex: HeadlessChromium
+  action: DENY

data/bots/us-ai-scraper.yaml

@@ -0,0 +1,3 @@
+- name: us-artificial-intelligence-scraper
+  user_agent_regex: \+https\://github\.com/US-Artificial-Intelligence/scraper
+  action: DENY

data/common/allow-private-addresses.yaml

@@ -0,0 +1,15 @@
+- name: ipv4-rfc-1918
+  action: ALLOW
+  remote_addresses:
+  - 10.0.0.0/8
+  - 172.16.0.0/12
+  - 192.168.0.0/16
+  - 100.64.0.0/10
+- name: ipv6-ula
+  action: ALLOW
+  remote_addresses:
+  - fc00::/7
+- name: ipv6-link-local
+  action: ALLOW
+  remote_addresses:
+  - fe80::/10

data/common/keep-internet-working.yaml

@@ -0,0 +1,10 @@
+# Common "keeping the internet working" routes
+- name: well-known
+  path_regex: ^/.well-known/.*$
+  action: ALLOW
+- name: favicon
+  path_regex: ^/favicon.ico$
+  action: ALLOW
+- name: robots-txt
+  path_regex: ^/robots.txt$
+  action: ALLOW

data/crawlers/bingbot.yaml

@@ -0,0 +1,34 @@
+- name: bingbot
+  user_agent_regex: \+http\://www\.bing\.com/bingbot\.htm
+  action: ALLOW
+  # https://www.bing.com/toolbox/bingbot.json
+  remote_addresses: [
+    "157.55.39.0/24",
+    "207.46.13.0/24",
+    "40.77.167.0/24",
+    "13.66.139.0/24",
+    "13.66.144.0/24",
+    "52.167.144.0/24",
+    "13.67.10.16/28",
+    "13.69.66.240/28",
+    "13.71.172.224/28",
+    "139.217.52.0/28",
+    "191.233.204.224/28",
+    "20.36.108.32/28",
+    "20.43.120.16/28",
+    "40.79.131.208/28",
+    "40.79.186.176/28",
+    "52.231.148.0/28",
+    "20.79.107.240/28",
+    "51.105.67.0/28",
+    "20.125.163.80/28",
+    "40.77.188.0/22",
+    "65.55.210.0/24",
+    "199.30.24.0/23",
+    "40.77.202.0/24",
+    "40.77.139.0/25",
+    "20.74.197.0/28",
+    "20.15.133.160/27",
+    "40.77.177.0/24",
+    "40.77.178.0/23"
+  ]

data/crawlers/duckduckbot.yaml

@@ -0,0 +1,275 @@
+- name: duckduckbot
+  user_agent_regex: DuckDuckBot/1\.1; \(\+http\://duckduckgo\.com/duckduckbot\.html\)
+  action: ALLOW
+  # https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot
+  remote_addresses: [
+    "57.152.72.128/32",
+    "51.8.253.152/32",
+    "40.80.242.63/32",
+    "20.12.141.99/32",
+    "20.49.136.28/32",
+    "51.116.131.221/32",
+    "51.107.40.209/32",
+    "20.40.133.240/32",
+    "20.50.168.91/32",
+    "51.120.48.122/32",
+    "20.193.45.113/32",
+    "40.76.173.151/32",
+    "40.76.163.7/32",
+    "20.185.79.47/32",
+    "52.142.26.175/32",
+    "20.185.79.15/32",
+    "52.142.24.149/32",
+    "40.76.162.208/32",
+    "40.76.163.23/32",
+    "40.76.162.191/32",
+    "40.76.162.247/32",
+    "40.88.21.235/32",
+    "20.191.45.212/32",
+    "52.146.59.12/32",
+    "52.146.59.156/32",
+    "52.146.59.154/32",
+    "52.146.58.236/32",
+    "20.62.224.44/32",
+    "51.104.180.53/32",
+    "51.104.180.47/32",
+    "51.104.180.26/32",
+    "51.104.146.225/32",
+    "51.104.146.235/32",
+    "20.73.202.147/32",
+    "20.73.132.240/32",
+    "20.71.12.143/32",
+    "20.56.197.58/32",
+    "20.56.197.63/32",
+    "20.43.150.93/32",
+    "20.43.150.85/32",
+    "20.44.222.1/32",
+    "40.89.243.175/32",
+    "13.89.106.77/32",
+    "52.143.242.6/32",
+    "52.143.241.111/32",
+    "52.154.60.82/32",
+    "20.197.209.11/32",
+    "20.197.209.27/32",
+    "20.226.133.105/32",
+    "191.234.216.4/32",
+    "191.234.216.178/32",
+    "20.53.92.211/32",
+    "20.53.91.2/32",
+    "20.207.99.197/32",
+    "20.207.97.190/32",
+    "40.81.250.205/32",
+    "40.64.106.11/32",
+    "40.64.105.247/32",
+    "20.72.242.93/32",
+    "20.99.255.235/32",
+    "20.113.3.121/32",
+    "52.224.16.221/32",
+    "52.224.21.53/32",
+    "52.224.20.204/32",
+    "52.224.21.19/32",
+    "52.224.20.249/32",
+    "52.224.20.203/32",
+    "52.224.20.190/32",
+    "52.224.16.229/32",
+    "52.224.21.20/32",
+    "52.146.63.80/32",
+    "52.224.20.227/32",
+    "52.224.20.193/32",
+    "52.190.37.160/32",
+    "52.224.21.23/32",
+    "52.224.20.223/32",
+    "52.224.20.181/32",
+    "52.224.21.49/32",
+    "52.224.21.55/32",
+    "52.224.21.61/32",
+    "52.224.19.152/32",
+    "52.224.20.186/32",
+    "52.224.21.27/32",
+    "52.224.21.51/32",
+    "52.224.20.174/32",
+    "52.224.21.4/32",
+    "51.104.164.109/32",
+    "51.104.167.71/32",
+    "51.104.160.177/32",
+    "51.104.162.149/32",
+    "51.104.167.95/32",
+    "51.104.167.54/32",
+    "51.104.166.111/32",
+    "51.104.167.88/32",
+    "51.104.161.32/32",
+    "51.104.163.250/32",
+    "51.104.164.189/32",
+    "51.104.167.19/32",
+    "51.104.160.167/32",
+    "51.104.167.110/32",
+    "20.191.44.119/32",
+    "51.104.167.104/32",
+    "20.191.44.234/32",
+    "51.104.164.215/32",
+    "51.104.167.52/32",
+    "20.191.44.22/32",
+    "51.104.167.87/32",
+    "51.104.167.96/32",
+    "20.191.44.16/32",
+    "51.104.167.61/32",
+    "51.104.164.147/32",
+    "20.50.48.159/32",
+    "40.114.182.172/32",
+    "20.50.50.130/32",
+    "20.50.50.163/32",
+    "20.50.50.46/32",
+    "40.114.182.153/32",
+    "20.50.50.118/32",
+    "20.50.49.55/32",
+    "20.50.49.25/32",
+    "40.114.183.251/32",
+    "20.50.50.123/32",
+    "20.50.49.237/32",
+    "20.50.48.192/32",
+    "20.50.50.134/32",
+    "51.138.90.233/32",
+    "40.114.183.196/32",
+    "20.50.50.146/32",
+    "40.114.183.88/32",
+    "20.50.50.145/32",
+    "20.50.50.121/32",
+    "20.50.49.40/32",
+    "51.138.90.206/32",
+    "40.114.182.45/32",
+    "51.138.90.161/32",
+    "20.50.49.0/32",
+    "40.119.232.215/32",
+    "104.43.55.167/32",
+    "40.119.232.251/32",
+    "40.119.232.50/32",
+    "40.119.232.146/32",
+    "40.119.232.218/32",
+    "104.43.54.127/32",
+    "104.43.55.117/32",
+    "104.43.55.116/32",
+    "104.43.55.166/32",
+    "52.154.169.50/32",
+    "52.154.171.70/32",
+    "52.154.170.229/32",
+    "52.154.170.113/32",
+    "52.154.171.44/32",
+    "52.154.172.2/32",
+    "52.143.244.81/32",
+    "52.154.171.87/32",
+    "52.154.171.250/32",
+    "52.154.170.28/32",
+    "52.154.170.122/32",
+    "52.143.243.117/32",
+    "52.143.247.235/32",
+    "52.154.171.235/32",
+    "52.154.171.196/32",
+    "52.154.171.0/32",
+    "52.154.170.243/32",
+    "52.154.170.26/32",
+    "52.154.169.200/32",
+    "52.154.170.96/32",
+    "52.154.170.88/32",
+    "52.154.171.150/32",
+    "52.154.171.205/32",
+    "52.154.170.117/32",
+    "52.154.170.209/32",
+    "191.235.202.48/32",
+    "191.233.3.202/32",
+    "191.235.201.214/32",
+    "191.233.3.197/32",
+    "191.235.202.38/32",
+    "20.53.78.144/32",
+    "20.193.24.10/32",
+    "20.53.78.236/32",
+    "20.53.78.138/32",
+    "20.53.78.123/32",
+    "20.53.78.106/32",
+    "20.193.27.215/32",
+    "20.193.25.197/32",
+    "20.193.12.126/32",
+    "20.193.24.251/32",
+    "20.204.242.101/32",
+    "20.207.72.113/32",
+    "20.204.242.19/32",
+    "20.219.45.67/32",
+    "20.207.72.11/32",
+    "20.219.45.190/32",
+    "20.204.243.55/32",
+    "20.204.241.148/32",
+    "20.207.72.110/32",
+    "20.204.240.172/32",
+    "20.207.72.21/32",
+    "20.204.246.81/32",
+    "20.207.107.181/32",
+    "20.204.246.254/32",
+    "20.219.43.246/32",
+    "52.149.25.43/32",
+    "52.149.61.51/32",
+    "52.149.58.139/32",
+    "52.149.60.38/32",
+    "52.148.165.38/32",
+    "52.143.95.162/32",
+    "52.149.56.151/32",
+    "52.149.30.45/32",
+    "52.149.58.173/32",
+    "52.143.95.204/32",
+    "52.149.28.83/32",
+    "52.149.58.69/32",
+    "52.148.161.87/32",
+    "52.149.58.27/32",
+    "52.149.28.18/32",
+    "20.79.226.26/32",
+    "20.79.239.66/32",
+    "20.79.238.198/32",
+    "20.113.14.159/32",
+    "20.75.144.152/32",
+    "20.43.172.120/32",
+    "20.53.134.160/32",
+    "20.201.15.208/32",
+    "20.93.28.24/32",
+    "20.61.34.40/32",
+    "52.242.224.168/32",
+    "20.80.129.80/32",
+    "20.195.108.47/32",
+    "4.195.133.120/32",
+    "4.228.76.163/32",
+    "4.182.131.108/32",
+    "4.209.224.56/32",
+    "108.141.83.74/32",
+    "4.213.46.14/32",
+    "172.169.17.165/32",
+    "51.8.71.117/32",
+    "20.3.1.178/32",
+    "52.149.56.151/32",
+    "52.149.30.45/32",
+    "52.149.58.173/32",
+    "52.143.95.204/32",
+    "52.149.28.83/32",
+    "52.149.58.69/32",
+    "52.148.161.87/32",
+    "52.149.58.27/32",
+    "52.149.28.18/32",
+    "20.79.226.26/32",
+    "20.79.239.66/32",
+    "20.79.238.198/32",
+    "20.113.14.159/32",
+    "20.75.144.152/32",
+    "20.43.172.120/32",
+    "20.53.134.160/32",
+    "20.201.15.208/32",
+    "20.93.28.24/32",
+    "20.61.34.40/32",
+    "52.242.224.168/32",
+    "20.80.129.80/32",
+    "20.195.108.47/32",
+    "4.195.133.120/32",
+    "4.228.76.163/32",
+    "4.182.131.108/32",
+    "4.209.224.56/32",
+    "108.141.83.74/32",
+    "4.213.46.14/32",
+    "172.169.17.165/32",
+    "51.8.71.117/32",
+    "20.3.1.178/32"
+  ]

data/crawlers/googlebot.yaml

@@ -0,0 +1,263 @@
+- name: googlebot
+  user_agent_regex: \+http\://www\.google\.com/bot\.html
+  action: ALLOW
+  # https://developers.google.com/static/search/apis/ipranges/googlebot.json
+  remote_addresses: [
+    "2001:4860:4801:10::/64",
+    "2001:4860:4801:11::/64",
+    "2001:4860:4801:12::/64",
+    "2001:4860:4801:13::/64",
+    "2001:4860:4801:14::/64",
+    "2001:4860:4801:15::/64",
+    "2001:4860:4801:16::/64",
+    "2001:4860:4801:17::/64",
+    "2001:4860:4801:18::/64",
+    "2001:4860:4801:19::/64",
+    "2001:4860:4801:1a::/64",
+    "2001:4860:4801:1b::/64",
+    "2001:4860:4801:1c::/64",
+    "2001:4860:4801:1d::/64",
+    "2001:4860:4801:1e::/64",
+    "2001:4860:4801:1f::/64",
+    "2001:4860:4801:20::/64",
+    "2001:4860:4801:21::/64",
+    "2001:4860:4801:22::/64",
+    "2001:4860:4801:23::/64",
+    "2001:4860:4801:24::/64",
+    "2001:4860:4801:25::/64",
+    "2001:4860:4801:26::/64",
+    "2001:4860:4801:27::/64",
+    "2001:4860:4801:28::/64",
+    "2001:4860:4801:29::/64",
+    "2001:4860:4801:2::/64",
+    "2001:4860:4801:2a::/64",
+    "2001:4860:4801:2b::/64",
+    "2001:4860:4801:2c::/64",
+    "2001:4860:4801:2d::/64",
+    "2001:4860:4801:2e::/64",
+    "2001:4860:4801:2f::/64",
+    "2001:4860:4801:31::/64",
+    "2001:4860:4801:32::/64",
+    "2001:4860:4801:33::/64",
+    "2001:4860:4801:34::/64",
+    "2001:4860:4801:35::/64",
+    "2001:4860:4801:36::/64",
+    "2001:4860:4801:37::/64",
+    "2001:4860:4801:38::/64",
+    "2001:4860:4801:39::/64",
+    "2001:4860:4801:3a::/64",
+    "2001:4860:4801:3b::/64",
+    "2001:4860:4801:3c::/64",
+    "2001:4860:4801:3d::/64",
+    "2001:4860:4801:3e::/64",
+    "2001:4860:4801:40::/64",
+    "2001:4860:4801:41::/64",
+    "2001:4860:4801:42::/64",
+    "2001:4860:4801:43::/64",
+    "2001:4860:4801:44::/64",
+    "2001:4860:4801:45::/64",
+    "2001:4860:4801:46::/64",
+    "2001:4860:4801:47::/64",
+    "2001:4860:4801:48::/64",
+    "2001:4860:4801:49::/64",
+    "2001:4860:4801:4a::/64",
+    "2001:4860:4801:4b::/64",
+    "2001:4860:4801:4c::/64",
+    "2001:4860:4801:50::/64",
+    "2001:4860:4801:51::/64",
+    "2001:4860:4801:52::/64",
+    "2001:4860:4801:53::/64",
+    "2001:4860:4801:54::/64",
+    "2001:4860:4801:55::/64",
+    "2001:4860:4801:56::/64",
+    "2001:4860:4801:60::/64",
+    "2001:4860:4801:61::/64",
+    "2001:4860:4801:62::/64",
+    "2001:4860:4801:63::/64",
+    "2001:4860:4801:64::/64",
+    "2001:4860:4801:65::/64",
+    "2001:4860:4801:66::/64",
+    "2001:4860:4801:67::/64",
+    "2001:4860:4801:68::/64",
+    "2001:4860:4801:69::/64",
+    "2001:4860:4801:6a::/64",
+    "2001:4860:4801:6b::/64",
+    "2001:4860:4801:6c::/64",
+    "2001:4860:4801:6d::/64",
+    "2001:4860:4801:6e::/64",
+    "2001:4860:4801:6f::/64",
+    "2001:4860:4801:70::/64",
+    "2001:4860:4801:71::/64",
+    "2001:4860:4801:72::/64",
+    "2001:4860:4801:73::/64",
+    "2001:4860:4801:74::/64",
+    "2001:4860:4801:75::/64",
+    "2001:4860:4801:76::/64",
+    "2001:4860:4801:77::/64",
+    "2001:4860:4801:78::/64",
+    "2001:4860:4801:79::/64",
+    "2001:4860:4801:80::/64",
+    "2001:4860:4801:81::/64",
+    "2001:4860:4801:82::/64",
+    "2001:4860:4801:83::/64",
+    "2001:4860:4801:84::/64",
+    "2001:4860:4801:85::/64",
+    "2001:4860:4801:86::/64",
+    "2001:4860:4801:87::/64",
+    "2001:4860:4801:88::/64",
+    "2001:4860:4801:90::/64",
+    "2001:4860:4801:91::/64",
+    "2001:4860:4801:92::/64",
+    "2001:4860:4801:93::/64",
+    "2001:4860:4801:94::/64",
+    "2001:4860:4801:95::/64",
+    "2001:4860:4801:96::/64",
+    "2001:4860:4801:a0::/64",
+    "2001:4860:4801:a1::/64",
+    "2001:4860:4801:a2::/64",
+    "2001:4860:4801:a3::/64",
+    "2001:4860:4801:a4::/64",
+    "2001:4860:4801:a5::/64",
+    "2001:4860:4801:c::/64",
+    "2001:4860:4801:f::/64",
+    "192.178.5.0/27",
+    "192.178.6.0/27",
+    "192.178.6.128/27",
+    "192.178.6.160/27",
+    "192.178.6.192/27",
+    "192.178.6.32/27",
+    "192.178.6.64/27",
+    "192.178.6.96/27",
+    "34.100.182.96/28",
+    "34.101.50.144/28",
+    "34.118.254.0/28",
+    "34.118.66.0/28",
+    "34.126.178.96/28",
+    "34.146.150.144/28",
+    "34.147.110.144/28",
+    "34.151.74.144/28",
+    "34.152.50.64/28",
+    "34.154.114.144/28",
+    "34.155.98.32/28",
+    "34.165.18.176/28",
+    "34.175.160.64/28",
+    "34.176.130.16/28",
+    "34.22.85.0/27",
+    "34.64.82.64/28",
+    "34.65.242.112/28",
+    "34.80.50.80/28",
+    "34.88.194.0/28",
+    "34.89.10.80/28",
+    "34.89.198.80/28",
+    "34.96.162.48/28",
+    "35.247.243.240/28",
+    "66.249.64.0/27",
+    "66.249.64.128/27",
+    "66.249.64.160/27",
+    "66.249.64.224/27",
+    "66.249.64.32/27",
+    "66.249.64.64/27",
+    "66.249.64.96/27",
+    "66.249.65.0/27",
+    "66.249.65.128/27",
+    "66.249.65.160/27",
+    "66.249.65.192/27",
+    "66.249.65.224/27",
+    "66.249.65.32/27",
+    "66.249.65.64/27",
+    "66.249.65.96/27",
+    "66.249.66.0/27",
+    "66.249.66.128/27",
+    "66.249.66.160/27",
+    "66.249.66.192/27",
+    "66.249.66.224/27",
+    "66.249.66.32/27",
+    "66.249.66.64/27",
+    "66.249.66.96/27",
+    "66.249.68.0/27",
+    "66.249.68.128/27",
+    "66.249.68.32/27",
+    "66.249.68.64/27",
+    "66.249.68.96/27",
+    "66.249.69.0/27",
+    "66.249.69.128/27",
+    "66.249.69.160/27",
+    "66.249.69.192/27",
+    "66.249.69.224/27",
+    "66.249.69.32/27",
+    "66.249.69.64/27",
+    "66.249.69.96/27",
+    "66.249.70.0/27",
+    "66.249.70.128/27",
+    "66.249.70.160/27",
+    "66.249.70.192/27",
+    "66.249.70.224/27",
+    "66.249.70.32/27",
+    "66.249.70.64/27",
+    "66.249.70.96/27",
+    "66.249.71.0/27",
+    "66.249.71.128/27",
+    "66.249.71.160/27",
+    "66.249.71.192/27",
+    "66.249.71.224/27",
+    "66.249.71.32/27",
+    "66.249.71.64/27",
+    "66.249.71.96/27",
+    "66.249.72.0/27",
+    "66.249.72.128/27",
+    "66.249.72.160/27",
+    "66.249.72.192/27",
+    "66.249.72.224/27",
+    "66.249.72.32/27",
+    "66.249.72.64/27",
+    "66.249.72.96/27",
+    "66.249.73.0/27",
+    "66.249.73.128/27",
+    "66.249.73.160/27",
+    "66.249.73.192/27",
+    "66.249.73.224/27",
+    "66.249.73.32/27",
+    "66.249.73.64/27",
+    "66.249.73.96/27",
+    "66.249.74.0/27",
+    "66.249.74.128/27",
+    "66.249.74.160/27",
+    "66.249.74.192/27",
+    "66.249.74.32/27",
+    "66.249.74.64/27",
+    "66.249.74.96/27",
+    "66.249.75.0/27",
+    "66.249.75.128/27",
+    "66.249.75.160/27",
+    "66.249.75.192/27",
+    "66.249.75.224/27",
+    "66.249.75.32/27",
+    "66.249.75.64/27",
+    "66.249.75.96/27",
+    "66.249.76.0/27",
+    "66.249.76.128/27",
+    "66.249.76.160/27",
+    "66.249.76.192/27",
+    "66.249.76.224/27",
+    "66.249.76.32/27",
+    "66.249.76.64/27",
+    "66.249.76.96/27",
+    "66.249.77.0/27",
+    "66.249.77.128/27",
+    "66.249.77.160/27",
+    "66.249.77.192/27",
+    "66.249.77.224/27",
+    "66.249.77.32/27",
+    "66.249.77.64/27",
+    "66.249.77.96/27",
+    "66.249.78.0/27",
+    "66.249.78.32/27",
+    "66.249.79.0/27",
+    "66.249.79.128/27",
+    "66.249.79.160/27",
+    "66.249.79.192/27",
+    "66.249.79.224/27",
+    "66.249.79.32/27",
+    "66.249.79.64/27",
+    "66.249.79.96/27"
+  ]

data/crawlers/internet-archive.yaml

@@ -0,0 +1,8 @@
+- name: internet-archive
+  action: ALLOW
+  # https://ipinfo.io/AS7941
+  remote_addresses: [
+    "207.241.224.0/20",
+    "208.70.24.0/21",
+    "2620:0:9c0::/48"
+  ]

data/crawlers/kagibot.yaml

@@ -0,0 +1,10 @@
+- name: kagibot
+  user_agent_regex: \+https\://kagi\.com/bot
+  action: ALLOW
+  # https://kagi.com/bot
+  remote_addresses: [
+    "216.18.205.234/32",
+    "35.212.27.76/32",
+    "104.254.65.50/32",
+    "209.151.156.194/32"
+  ]

data/crawlers/marginalia.yaml

@@ -0,0 +1,11 @@
+- name: marginalia
+  user_agent_regex: search\.marginalia\.nu
+  action: ALLOW
+  # Received directly over email
+  remote_addresses: [
+    "193.183.0.162/31",
+    "193.183.0.164/30",
+    "193.183.0.168/30",
+    "193.183.0.172/31",
+    "193.183.0.174/32"
+  ]

data/crawlers/mojeekbot.yaml

@@ -0,0 +1,5 @@
+- name: mojeekbot
+  user_agent_regex: \+https\://www\.mojeek\.com/bot\.html
+  action: ALLOW
+  # https://www.mojeek.com/bot.html
+  remote_addresses: [ "5.102.173.71/32" ]

data/crawlers/qwantbot.yaml

@@ -0,0 +1,5 @@
+- name: qwantbot
+  user_agent_regex: \+https\://help\.qwant\.com/bot/
+  action: ALLOW
+  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
+  remote_addresses: [ "91.242.162.0/24" ]

data/embed.go

@@ -3,6 +3,6 @@ package data
 import "embed"
 
 var (
-	//go:embed botPolicies.json
+	//go:embed botPolicies.yaml botPolicies.json apps bots common crawlers
 	BotPolicies embed.FS
 )

docs/docs/CHANGELOG.md

@@ -11,17 +11,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+- Ensure regexes can't end in newlines ([#372](https://github.com/TecharoHQ/anubis/issues/372))
+- Add documentation for default allow behavior (implicit rule)
+- Enable [importing configuration snippets](./admin/configuration/import.mdx) ([#321](https://github.com/TecharoHQ/anubis/pull/321))
+- Refactor check logic to be more generic and work on a Checker type
 - Add more AI user agents based on the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) project
 - Embedded challenge data in initial HTML response to improve performance
+- Added support to use Nginx' `auth_request` directive with Anubis
+- Added support to allow to restrict the allowed redirect domains
 - Whitelisted [DuckDuckBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/) in botPolicies
 - Improvements to build scripts to make them less independent of the build host
 - Improved the OpenGraph error logging
 - Added `Opera` to the `generic-browser` bot policy rule
-- Added FreeBSD rc.d script so can be run as a FreeBSD daemon.
+- Added FreeBSD rc.d script so can be run as a FreeBSD daemon
 - Allow requests from the Internet Archive
 - Added example nginx configuration to documentation
 - Added example Apache configuration to the documentation [#277](https://github.com/TecharoHQ/anubis/issues/277)
 - Move per-environment configuration details into their own pages
+- Added support for running anubis behind a prefix (e.g. `/myapp`)
+- Added headers support to bot policy rules
+- Moved configuration file from JSON to YAML by default
+- Added documentation on how to use Anubis with Traefik in Docker
+- Improved error handling in some edge cases
+- Disable `generic-bot-catchall` rule because of its high false positive rate in real-world scenarios
+- Moved all CSS inline to the Xess package, changed colors to be CSS variables
+- Set or append to `X-Forwarded-For` header unless the remote connects over a loopback address [#328](https://github.com/TecharoHQ/anubis/issues/328)
+- Fixed mojeekbot user agent regex
+- Added support for running anubis behind a base path (e.g. `/myapp`)
+- Reduce Anubis' paranoia with user cookies ([#365](https://github.com/TecharoHQ/anubis/pull/365))
 
 ## v1.16.0
 
@@ -31,39 +48,39 @@ Fordola rem Lupis
 
 The following features are the "big ticket" items:
 
-- Added support for native Debian, Red Hat, and tarball packaging strategies including installation and use directions.
-- A prebaked tarball has been added, allowing distros to build Anubis like they could in v1.15.x.
-- The placeholder Anubis mascot has been replaced with a design by [CELPHASE](https://bsky.app/profile/celphase.bsky.social).
-- Verification page now shows hash rate and a progress bar for completion probability.
+- Added support for native Debian, Red Hat, and tarball packaging strategies including installation and use directions
+- A prebaked tarball has been added, allowing distros to build Anubis like they could in v1.15.x
+- The placeholder Anubis mascot has been replaced with a design by [CELPHASE](https://bsky.app/profile/celphase.bsky.social)
+- Verification page now shows hash rate and a progress bar for completion probability
 - Added support for [OpenGraph tags](https://ogp.me/) when rendering the challenge page. This allows for social previews to be generated when sharing the challenge page on social media platforms ([#195](https://github.com/TecharoHQ/anubis/pull/195))
-- Added support for passing the ed25519 signing key in a file with `-ed25519-private-key-hex-file` or `ED25519_PRIVATE_KEY_HEX_FILE`.
+- Added support for passing the ed25519 signing key in a file with `-ed25519-private-key-hex-file` or `ED25519_PRIVATE_KEY_HEX_FILE`
 
 The other small fixes have been made:
 
-- Added a periodic cleanup routine for the decaymap that removes expired entries, ensuring stale data is properly pruned.
+- Added a periodic cleanup routine for the decaymap that removes expired entries, ensuring stale data is properly pruned
 - Added a no-store Cache-Control header to the challenge page
 - Hide the directory listings for Anubis' internal static content
-- Changed `--debug-x-real-ip-default` to `--use-remote-address`, getting the IP address from the request's socket address instead.
+- Changed `--debug-x-real-ip-default` to `--use-remote-address`, getting the IP address from the request's socket address instead
 - DroneBL lookups have been disabled by default
 - Static asset builds are now done on demand instead of the results being committed to source control
 - The Dockerfile has been removed as it is no longer in use
 - Developer documentation has been added to the docs site
 - Show more errors when some predictable challenge page errors happen ([#150](https://github.com/TecharoHQ/anubis/issues/150))
-- Added the `--debug-benchmark-js` flag for testing proof-of-work performance during development.
+- Added the `--debug-benchmark-js` flag for testing proof-of-work performance during development
 - Use `TrimSuffix` instead of `TrimRight` on containerbuild
 - Fix the startup logs to correctly show the address and port the server is listening on
 - Add [LibreJS](https://www.gnu.org/software/librejs/) banner to Anubis JavaScript to allow LibreJS users to run the challenge
 - Added a wait with button continue + 30 second auto continue after 30s if you click "Why am I seeing this?"
-- Fixed a typo in the challenge page title.
-- Disabled running integration tests on Windows hosts due to it's reliance on posix features (see [#133](https://github.com/TecharoHQ/anubis/pull/133#issuecomment-2764732309)).
+- Fixed a typo in the challenge page title
+- Disabled running integration tests on Windows hosts due to it's reliance on posix features (see [#133](https://github.com/TecharoHQ/anubis/pull/133#issuecomment-2764732309))
 - Fixed minor typos
-- Added a Makefile to enable comfortable workflows for downstream packagers.
+- Added a Makefile to enable comfortable workflows for downstream packagers
 - Added `zizmor` for GitHub Actions static analysis
 - Fixed most `zizmor` findings
 - Enabled Dependabot
 - Added an air config for autoreload support in development ([#195](https://github.com/TecharoHQ/anubis/pull/195))
-- Added an `--extract-resources` flag to extract static resources to a local folder.
-- Add noindex flag to all Anubis pages ([#227](https://github.com/TecharoHQ/anubis/issues/227)).
+- Added an `--extract-resources` flag to extract static resources to a local folder
+- Add noindex flag to all Anubis pages ([#227](https://github.com/TecharoHQ/anubis/issues/227))
 - Added `WEBMASTER_EMAIL` variable, if it is present then display that email address on error pages ([#235](https://github.com/TecharoHQ/anubis/pull/235), [#115](https://github.com/TecharoHQ/anubis/issues/115))
 - Hash pinned all GitHub Actions
 
@@ -148,7 +165,7 @@ Livia sas Junius
   [#21](https://github.com/TecharoHQ/anubis/pull/21)
 - Don't overflow the image when browser windows are small (eg. on phones)
   [#27](https://github.com/TecharoHQ/anubis/pull/27)
-- Lower the default difficulty to 4 from 5
+- Lower the default difficulty to 5 from 4
 - Don't duplicate work across multiple threads [#36](https://github.com/TecharoHQ/anubis/pull/36)
 - Documentation has been moved to https://anubis.techaro.lol/ with sources in docs/
 - Removed several visible AI artifacts (e.g., 6 fingers) [#37](https://github.com/TecharoHQ/anubis/pull/37)
@@ -191,4 +208,4 @@ Livia sas Junius
   ([fd6903a](https://github.com/TecharoHQ/anubis/commit/fd6903aeed315b8fddee32890d7458a9271e4798)).
 - Footer links on the check page now point to Techaro's brand
   ([4ebccb1](https://github.com/TecharoHQ/anubis/commit/4ebccb197ec20d024328d7f92cad39bbbe4d6359))
-- Anubis was imported from [Xe/x](https://github.com/Xe/x).
+- Anubis was imported from [Xe/x](https://github.com/Xe/x)

docs/docs/admin/configuration/import.mdx

@@ -0,0 +1,147 @@
+# Importing configuration rules
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis has the ability to let you import snippets of configuration into the main configuration file. This allows you to break up your config into smaller parts that get logically assembled into one big file.
+
+EG:
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+{
+  "bots": [
+    {
+      "import": "(data)/bots/ai-robots-txt.yaml"
+    },
+    {
+      "import": "(data)/bots/cloudflare-workers.yaml"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+bots:
+  # Pathological bots to deny
+  - # This correlates to data/bots/ai-robots-txt.yaml in the source tree
+    import: (data)/bots/ai-robots-txt.yaml
+  - import: (data)/bots/cloudflare-workers.yaml
+```
+
+</TabItem>
+</Tabs>
+
+Of note, a bot rule can either have inline bot configuration or import a bot config snippet. You cannot do both in a single bot rule.
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+{
+  "bots": [
+    {
+      "import": "(data)/bots/ai-robots-txt.yaml",
+      "name": "generic-browser",
+      "user_agent_regex": "Mozilla|Opera\n",
+      "action": "CHALLENGE"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+bots:
+  - import: (data)/bots/ai-robots-txt.yaml
+    name: generic-browser
+    user_agent_regex: >
+      Mozilla|Opera
+    action: CHALLENGE
+```
+
+</TabItem>
+</Tabs>
+
+This will return an error like this:
+
+```text
+config is not valid:
+config.BotOrImport: rule definition is invalid, you must set either bot rules or an import statement, not both
+```
+
+Paths can either be prefixed with `(data)` to import from the [the data folder in the Anubis source tree](https://github.com/TecharoHQ/anubis/tree/main/data) or anywhere on the filesystem. If you don't have access to the Anubis source tree, check /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
+
+## Writing snippets
+
+Snippets can be written in either JSON or YAML, with a preference for YAML. When writing a snippet, write the bot rules you want directly at the top level of the file in a list.
+
+Here is an example snippet that allows [IPv6 Unique Local Addresses](https://en.wikipedia.org/wiki/Unique_local_address) through Anubis:
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+[
+  {
+    "name": "ipv6-ula",
+    "action": "ALLOW",
+    "remote_addresses": ["fc00::/7"]
+  }
+]
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+- name: ipv6-ula
+  action: ALLOW
+  remote_addresses:
+    - fc00::/7
+```
+
+</TabItem>
+</Tabs>
+
+## Extracting Anubis' embedded filesystem
+
+You can always extract the list of rules embedded into the Anubis binary with this command:
+
+```text
+anubis --extract-resources=static
+```
+
+This will dump the contents of Anubis' embedded data to a new folder named `static`:
+
+```text
+static
+├── apps
+│   └── gitea-rss-feeds.yaml
+├── botPolicies.json
+├── botPolicies.yaml
+├── bots
+│   ├── ai-robots-txt.yaml
+│   ├── cloudflare-workers.yaml
+│   ├── headless-browsers.yaml
+│   └── us-ai-scraper.yaml
+├── common
+│   ├── allow-private-addresses.yaml
+│   └── keep-internet-working.yaml
+└── crawlers
+    ├── bingbot.yaml
+    ├── duckduckbot.yaml
+    ├── googlebot.yaml
+    ├── internet-archive.yaml
+    ├── kagibot.yaml
+    ├── marginalia.yaml
+    ├── mojeekbot.yaml
+    └── qwantbot.yaml
+```

docs/docs/admin/configuration/redirect-domains.mdx

@@ -0,0 +1,94 @@
+---
+title: Redirect Domain Configuration
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis has an HTTP redirect in the middle of its check validation logic. This redirect allows Anubis to set a cookie on validated requests so that users don't need to pass challenges on every page load.
+
+This flow looks something like this:
+
+```mermaid
+sequenceDiagram
+  participant User
+  participant Challenge
+  participant Validation
+  participant Backend
+
+  User->>+Challenge: GET /
+  Challenge->>+User: Solve this challenge
+  User->>+Validation: Here's the solution, send me to /
+  Validation->>+User: Here's a cookie, go to /
+  User->>+Backend: GET /
+```
+
+However, in some cases a sufficiently dedicated attacker could trick a user into clicking on a validation link with a solution pre-filled out. For example:
+
+```mermaid
+sequenceDiagram
+  participant Hacker
+  participant User
+  participant Validation
+  participant Evil Site
+
+  Hacker->>+User: Click on yoursite.com with this solution
+  User->>+Validation: Here's a solution, send me to evilsite.com
+  Validation->>+User: Here's a cookie, go to evilsite.com
+  User->>+Evil Site: GET evilsite.com
+```
+
+If this happens, Anubis will throw an error like this:
+
+```text
+Redirect domain not allowed
+```
+
+## Configuring allowed redirect domains
+
+By default, Anubis will limit redirects to be on the same HTTP Host that Anubis is running on (EG: requests to yoursite.com cannot redirect outside of yoursite.com). If you need to set more than one domain, fill the `REDIRECT_DOMAINS` environment variable with a comma-separated list of domain names that Anubis should allow redirects to.
+
+:::note
+
+These domains are _an exact string match_, they do not support wildcard matches.
+
+:::
+
+<Tabs>
+  <TabItem value="env-file" label="Environment file" default>
+
+```shell
+# anubis.env
+
+REDIRECT_DOMAINS="yoursite.com,secretplans.yoursite.com"
+# ...
+```
+
+  </TabItem>
+  <TabItem value="docker-compose" label="Docker Compose">
+
+```yaml
+services:
+  anubis-nginx:
+    image: ghcr.io/techarohq/anubis:latest
+    environment:
+      REDIRECT_DOMAINS: "yoursite.com,secretplans.yoursite.com"
+      # ...
+```
+
+  </TabItem>
+  <TabItem value="k8s" label="Kubernetes">
+
+Inside your Deployment, StatefulSet, or Pod:
+
+```yaml
+- name: anubis
+  image: ghcr.io/techarohq/anubis:latest
+  env:
+    - name: REDIRECT_DOMAINS
+      value: "yoursite.com,secretplans.yoursite.com"
+    # ...
+```
+
+  </TabItem>
+</Tabs>

docs/docs/admin/configuration/subrequest-auth.mdx

@@ -0,0 +1,139 @@
+---
+title: Subrequest Authentication
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis can act in one of two modes:
+
+1. Reverse proxy (the default): Anubis sits in the middle of all traffic and then will reverse proxy it to its destination. This is the moral equivalent of a middleware in your favorite web framework.
+2. Subrequest authentication mode: Anubis listens for requests and if they don't pass muster then they are forwarded to Anubis for challenge processing. This is the equivalent of Anubis being a sidecar service.
+
+## Nginx
+
+Anubis can perform [subrequest authentication](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/) with the `auth_request` module in Nginx. In order to set this up, keep the following things in mind:
+
+The `TARGET` environment variable in Anubis must be set to a space, eg:
+
+<Tabs>
+  <TabItem value="env-file" label="Environment file" default>
+
+```shell
+# anubis.env
+
+TARGET=" "
+# ...
+```
+
+  </TabItem>
+  <TabItem value="docker-compose" label="Docker Compose">
+
+```yaml
+services:
+  anubis-nginx:
+    image: ghcr.io/techarohq/anubis:latest
+    environment:
+      TARGET: " "
+      # ...
+```
+
+  </TabItem>
+  <TabItem value="k8s" label="Kubernetes">
+
+Inside your Deployment, StatefulSet, or Pod:
+
+```yaml
+- name: anubis
+  image: ghcr.io/techarohq/anubis:latest
+  env:
+    - name: TARGET
+      value: " "
+    # ...
+```
+
+  </TabItem>
+</Tabs>
+
+In order to configure this, you need to add the following location blocks to each server pointing to the service you want to protect:
+
+```nginx
+location /.within.website/ {
+    # Assumption: Anubis is running in the same network namespace as
+    # nginx on localhost TCP port 8923
+    proxy_pass http://127.0.0.1:8923;
+    auth_request off;
+}
+
+location @redirectToAnubis {
+    return 307 /.within.website/?redir=$scheme://$host$request_uri;
+    auth_request off;
+}
+```
+
+This sets up `/.within.website` to point to Anubis. Any requests that Anubis rejects or throws a challenge to will be sent here. This also sets up a named location `@redirectToAnubis` that will redirect any requests to Anubis for advanced processing.
+
+Finally, add this to your root location block:
+
+```nginx
+location / {
+    # diff-add
+    auth_request /.within.website/x/cmd/anubis/api/check;
+    # diff-add
+    error_page 401 = @redirectToAnubis;
+}
+```
+
+This will check all requests that don't match other locations with Anubis to ensure the client is genuine.
+
+This will make every request get checked by Anubis before it hits your backend. If you have other locations that don't need Anubis to do validation, add the `auth_request off` directive to their blocks:
+
+```nginx
+location /secret {
+    # diff-add
+    auth_request off;
+
+    # ...
+}
+```
+
+Here is a complete example of an Nginx server listening over TLS and pointing to Anubis:
+
+<details>
+  <summary>Complete example</summary>
+
+```nginx
+# /etc/nginx/conf.d/nginx.local.cetacean.club.conf
+
+server {
+  listen 443 ssl;
+  listen [::]:443 ssl;
+  server_name         nginx.local.cetacean.club;
+  ssl_certificate     /etc/techaro/pki/nginx.local.cetacean.club/tls.crt;
+  ssl_certificate_key /etc/techaro/pki/nginx.local.cetacean.club/tls.key;
+  ssl_protocols       TLSv1.2 TLSv1.3;
+  ssl_ciphers         HIGH:!aNULL:!MD5;
+
+  proxy_set_header X-Real-IP $remote_addr;
+  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+  location /.within.website/ {
+    proxy_pass http://localhost:8923;
+    auth_request off;
+  }
+
+  location @redirectToAnubis {
+    return 307 /.within.website/?redir=$scheme://$host$request_uri;
+    auth_request off;
+  }
+
+  location / {
+    auth_request /.within.website/x/cmd/anubis/api/check;
+    error_page 401 = @redirectToAnubis;
+    root /usr/share/nginx/html;
+    index index.html index.htm;
+  }
+}
+```
+
+</details>

docs/docs/admin/default-allow-behavior.mdx

@@ -0,0 +1,92 @@
+---
+title: Default allow behavior
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+# Default allow behavior
+
+Anubis is designed to be as unintrusive as possible to your existing infrastructure.
+
+By default, it allows all traffic unless a request matches a rule that explicitly denies or challenges it.
+
+Only requests matching a DENY or CHALLENGE rule are blocked or challenged. All other requests are allowed. This is called "the implicit rule".
+
+## Example: Minimal policy
+
+If your policy only blocks a specific bot, all other requests will be allowed:
+
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+```json
+{
+  "bots": [
+    {
+      "name": "block-amazonbot",
+      "user_agent_regex": "Amazonbot",
+      "action": "DENY"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: block-amazonbot
+  user_agent_regex: Amazonbot
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
+## How to deny by default
+
+If you want to deny all traffic except what you explicitly allow, add a catch-all deny rule at the end of your policy list. Make sure to add ALLOW rules for any traffic you want to permit before this rule.
+
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+```json
+{
+  "bots": [
+    {
+      "name": "allow-goodbot",
+      "user_agent_regex": "GoodBot",
+      "action": "ALLOW"
+    },
+    {
+      "name": "catch-all-deny",
+      "path_regex": ".*",
+      "action": "DENY"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: allow-goodbot
+  user_agent_regex: GoodBot
+  action: ALLOW
+- name: catch-all-deny
+  path_regex: .*
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
+## Final remarks
+
+- Rules are evaluated in order; the first match wins.
+- The implicit allow rule is always last and cannot be removed.
+- Use your logs to monitor what traffic is being allowed by default.
+
+See [Policy Definitions](./policies) for more details on writing rules.

docs/docs/admin/environments/docker-compose.mdx

@@ -8,17 +8,17 @@ services:
     image: ghcr.io/techarohq/anubis:latest
     environment:
       BIND: ":8080"
-      DIFFICULTY: "5"
+      DIFFICULTY: "4"
       METRICS_BIND: ":9090"
       SERVE_ROBOTS_TXT: "true"
       TARGET: "http://nginx"
-      POLICY_FNAME: "/data/cfg/botPolicy.json"
+      POLICY_FNAME: "/data/cfg/botPolicy.yaml"
       OG_PASSTHROUGH: "true"
       OG_EXPIRY_TIME: "24h"
     ports:
       - 8080:8080
     volumes:
-      - "./botPolicy.json:/data/cfg/botPolicy.json:ro"
+      - "./botPolicy.yaml:/data/cfg/botPolicy.yaml:ro"
   nginx:
     image: nginx
     volumes:

docs/docs/admin/environments/nginx.mdx

@@ -41,45 +41,45 @@ Assuming that we are protecting `anubistest.techaro.lol`, here's what the server
 
 # HTTP - Redirect all HTTP traffic to HTTPS
 server {
-	listen 80;
-	listen [::]:80;
+  listen 80;
+  listen [::]:80;
 
-	server_name anubistest.techaro.lol;
+  server_name anubistest.techaro.lol;
 
-	location / {
-		return 301 https://$host$request_uri;
-	}
+  location / {
+    return 301 https://$host$request_uri;
+  }
 }
 
 # TLS termination server, this will listen over TLS (https) and then
 # proxy all traffic to the target via Anubis.
 server {
-	# Listen on TCP port 443 with TLS (https) and HTTP/2
-	listen 443 ssl http2;
-	listen [::]:443 ssl http2;
+  # Listen on TCP port 443 with TLS (https) and HTTP/2
+  listen 443 ssl http2;
+  listen [::]:443 ssl http2;
 
-	location / {
+  location / {
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_pass http://anubis;
   }
 
-	server_name anubistest.techaro.lol;
+  server_name anubistest.techaro.lol;
 
-	ssl_certificate      /path/to/your/certs/anubistest.techaro.lol.crt;
-	ssl_certificate_key	 /path/to/your/certs/anubistest.techaro.lol.key;
+  ssl_certificate     /path/to/your/certs/anubistest.techaro.lol.crt;
+  ssl_certificate_key /path/to/your/certs/anubistest.techaro.lol.key;
 }
 
 # Backend server, this is where your webapp should actually live.
 server {
-	listen unix:/run/nginx/nginx.sock;
+  listen unix:/run/nginx/nginx.sock;
 
-	server_name anubistest.techaro.lol;
-	root "/srv/http/anubistest.techaro.lol";
-	index index.html;
+  server_name anubistest.techaro.lol;
+  root "/srv/http/anubistest.techaro.lol";
+  index index.html;
 
-	# Your normal configuration can go here
-	# location .php { fastcgi...} etc.
+  # Your normal configuration can go here
+  # location .php { fastcgi...} etc.
 }
 ```
 
@@ -107,28 +107,28 @@ Then in a server block:
 # /etc/nginx/conf.d/server-mimi-techaro-lol.conf
 
 server {
-	# Listen on 443 with SSL
-	listen 443 ssl http2;
-	listen [::]:443 ssl http2;
+  # Listen on 443 with SSL
+  listen 443 ssl http2;
+  listen [::]:443 ssl http2;
 
-	# Slipstream via Anubis
-	include "conf-anubis.inc";
+  # Slipstream via Anubis
+  include "conf-anubis.inc";
 
-	server_name mimi.techaro.lol;
+  server_name mimi.techaro.lol;
 
-	ssl_certificate		   /path/to/your/certs/mimi.techaro.lol.crt;
-	ssl_certificate_key	 /path/to/your/certs/mimi.techaro.lol.key;
+  ssl_certificate     /path/to/your/certs/mimi.techaro.lol.crt;
+  ssl_certificate_key /path/to/your/certs/mimi.techaro.lol.key;
 }
 
 server {
-	listen unix:/run/nginx/nginx.sock;
+  listen unix:/run/nginx/nginx.sock;
 
-	server_name mimi.techaro.lol;
-	root "/srv/http/mimi.techaro.lol";
-	index index.html;
+  server_name mimi.techaro.lol;
+  root "/srv/http/mimi.techaro.lol";
+  index index.html;
 
-	# Your normal configuration can go here
-	# location .php { fastcgi...} etc.
+  # Your normal configuration can go here
+  # location .php { fastcgi...} etc.
 }
 ```
 

docs/docs/admin/environments/traefik.mdx

@@ -0,0 +1,215 @@
+---
+id: traefik
+title: Integrate Anubis with Traefik in a Docker Compose Environment
+---
+
+
+:::note
+
+ This only talks about integration through Compose,
+ but it also applies to docker cli options.
+
+:::
+
+Currently, Anubis doesn't have any Traefik middleware,
+so you need to manually route it between Traefik and your target service.
+This routing is done per labels in Traefik.
+
+In this example, we will use 4 Containers:
+
+- `traefik` - the Traefik instance
+- `anubis` - the Anubis instance
+- `target` - our service to protect (`traefik/whoami` in this case)
+- `target2` - a second service that isn't supposed to be protected (`traefik/whoami` in this case)
+
+There are 3 steps we need to follow:
+
+1. Create a new exclusive Traefik endpoint for Anubis
+2. Pass all unspecified requests to Anubis
+3. Let Anubis pass all verified requests back to Traefik on its exclusive endpoint
+
+## Diagram of Flow
+
+This is a small diagram depicting the flow.
+Keep in mind that `8080` or `80` can be anything depending on your containers.
+
+```mermaid
+flowchart LR
+user[User]
+traefik[Traefik]
+anubis[Anubis]
+target[Target]
+
+user-->|:443 - Requesting Service|traefik
+traefik-->|:8080 - Passing to Anubis|anubis
+anubis-->|:3923 - Passing back to Traefik|traefik
+traefik-->|:80 - Passing to the target|target
+```
+
+## Create an Exclusive Anubis Endpoint in Traefik
+
+There are 2 ways of registering a new endpoint in Traefik.
+Which one to use depends on how you configured your Traefik so far.
+
+**CLI Options:**
+
+```yml
+--entrypoints.anubis.address=:3923
+```
+
+**traefik.yml:**
+
+```yml
+entryPoints:
+  anubis:
+    address: ":3923"
+```
+
+It is important that the specified port isn't actually reachable from the outside,
+but only exposed in the Docker network.
+Exposing the Anubis port on Traefik directly will allow direct unprotected access to all containers behind it.
+
+## Passing all unspecified Web Requests to Anubis
+
+There are cases where you want Traefik to still route some requests without protection, just like before.
+To achieve this, we can register Anubis as the default handler for non-protected requests.
+
+We also don't want users to get SSL Errors during the checking phase,
+thus we also need to let Traefik provide SSL Certs for our endpoint.
+This example expects an TLS cert resolver called `le`.
+
+We also expect there to be an endpoint called `websecure` for HTTPS in this example.
+
+This is an example of the required labels to configure Traefik on the Anubis container:
+
+```yml
+labels:
+  - traefik.enable=true # Enabling Traefik
+  - traefik.docker.network=traefik # Telling Traefik which network to use
+  - traefik.http.routers.anubis.priority=1 # Setting Anubis to the lowest priority, so it only takes the slack
+  - traefik.http.routers.anubis.rule=PathRegexp(`.*`) # Wildcard match every path
+  - traefik.http.routers.anubis.entrypoints=websecure # Listen on HTTPS
+  - traefik.http.services.anubis.loadbalancer.server.port=8080 # Telling Traefik to which port it should route requests
+  - traefik.http.routers.anubis.service=anubis # Telling Traefik to use the above specified port
+  - traefik.http.routers.anubis.tls.certresolver=le # Telling Traefik to resolve a Cert for Anubis
+```
+
+## Passing all Verified Requests Back Correctly to Traefik
+
+To pass verified requests back to Traefik,
+we only need to configure Anubis using its environment variables:
+
+```yml
+environment:
+  - BIND=:8080
+  - TARGET=http://traefik:3923
+```
+
+## Full Example Config
+
+Now that we know how to pass all requests back and forth, here is the example.
+This example contains 2 services: one that is protected and the other one that is not.
+
+**compose.yml**
+
+```yml
+services:
+  traefik:
+    image: traefik:v3.3
+    ports:
+      - 80:80
+      - 443:443
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./letsencrypt:/letsencrypt
+      - ./traefik.yml:/traefik.yml:ro
+    networks:
+      - traefik
+    labels:
+      # Enable Traefik
+      - traefik.enable=true
+      - traefik.docker.network=traefik
+      # Redirect any HTTP to HTTPS
+      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
+      - traefik.http.routers.web.rule=PathPrefix(`/`)
+      - traefik.http.routers.web.entrypoints=web
+      - traefik.http.routers.web.middlewares=redirect-to-https
+      - traefik.http.routers.web.tls=false
+
+  anubis:
+    image: ghcr.io/techarohq/anubis:main
+    environment:
+      # Telling Anubis, where to listen for Traefik
+      - BIND=:8080
+      # Telling Anubis to point to Traefik via the Docker network
+      - TARGET=http://traefik:3923
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Enabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.anubis.priority=1 # Setting Anubis to the lowest priority, so it only takes the slack
+      - traefik.http.routers.anubis.rule=PathRegexp(`.*`) # wildcard match anything
+      - traefik.http.routers.anubis.entrypoints=websecure # Listen on HTTPS
+      - traefik.http.services.anubis.loadbalancer.server.port=8080 # Telling Traefik to which port it should route requests
+      - traefik.http.routers.anubis.service=anubis # Telling Traefik to use the above specified port
+      - traefik.http.routers.anubis.tls.certresolver=le # Telling Traefik to resolve a Cert for Anubis
+
+  # Protected by Anubis
+  target:
+    image: traefik/whoami:latest
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Enabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.target.rule=Host(`example.com`) # Only Matching Requests for example.com
+      - traefik.http.routers.target.entrypoints=anubis # Listening on the exclusive Anubis Network
+      - traefik.http.services.target.loadbalancer.server.port=80 # Telling Traefik where to receive requests
+      - traefik.http.routers.target.service=target # Telling Traefik to use the above specified port
+
+  # Not Protected by Anubis
+  target2:
+    image: traefik/whoami:latest
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Eneabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.target2.rule=Host(`another.com`) # Only Matching Requests for example.com
+      - traefik.http.routers.target2.entrypoints=websecure # Listening on the exclusive Anubis Network
+      - traefik.http.services.target2.loadbalancer.server.port=80 # Telling Traefik where to receive requests
+      - traefik.http.routers.target2.service=target2 # Telling Traefik to use the above specified port
+      - traefik.http.routers.target2.tls.certresolver=le # Telling Traefik to resolve a Cert for this Target
+
+networks:
+  traefik:
+    name: traefik
+```
+
+**traefik.yml**
+
+```yml
+api:
+  insecure: false # shouldn't be enabled in prod
+
+entryPoints:
+  # Web
+  web:
+    address: ":80"
+  websecure:
+    address: ":443"
+  # Anubis
+  anubis:
+    address: ":3923"
+
+certificatesResolvers:
+  le:
+    acme:
+      tlsChallenge: {}
+      email: "admin@example.com"
+      storage: "/letsencrypt/acme.json"
+
+providers:
+  docker: {}
+```

docs/docs/admin/installation.mdx

@@ -49,28 +49,66 @@ For more detailed information on installing Anubis with native packages, please
 
 Anubis uses these environment variables for configuration:
 
-| Environment Variable           | Default value           | Explanation                                                                                                                                                                                                                                                                              |
-| :----------------------------- | :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `BIND`                         | `:8923`                 | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock`                                                                                                                                                                                  |
-| `BIND_NETWORK`                 | `tcp`                   | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports.                                                                                                                                               |
-| `COOKIE_DOMAIN`                | unset                   | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See [here](https://stackoverflow.com/a/1063760) for more information.               |
-| `COOKIE_PARTITIONED`           | `false`                 | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe.                                                           |
-| `DIFFICULTY`                   | `5`                     | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses.                                                                                                                                                                                   |
-| `ED25519_PRIVATE_KEY_HEX`      | unset                   | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details.                                                                                               |
-| `ED25519_PRIVATE_KEY_HEX_FILE` | unset                   | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set.                                                                                                                                                                         |
-| `METRICS_BIND`                 | `:9090`                 | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information.                                                                                                                                                                                           |
-| `METRICS_BIND_NETWORK`         | `tcp`                   | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information.                                                                                                                                                                                   |
-| `OG_EXPIRY_TIME`               | `24h`                   | The expiration time for the Open Graph tag cache.                                                                                                                                                                                                                                        |
-| `OG_PASSTHROUGH`               | `false`                 | If set to `true`, Anubis will enable Open Graph tag passthrough.                                                                                                                                                                                                                         |
-| `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.md). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                |
-| `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file. |
-| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                |
-| `TARGET`                       | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`.                                                                                                                             |
-| `USE_REMOTE_ADDRESS`           | unset                   | If set to `true`, Anubis will take the client's IP from the network socket. For production deployments, it is expected that a reverse proxy is used in front of Anubis, which pass the IP using headers, instead.                                                                        |
-| `WEBMASTER_EMAIL`              | unset                   | If set, shows a contact email address when rendering error pages. This email address will be how users can get in contact with administrators.                                                                                                                                           |
+| Environment Variable           | Default value           | Explanation                                                                                                                                                                                                                                                                                                          |
+| :----------------------------- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `BASE_PREFIX`                  | unset                   | If set, adds a global prefix to all Anubis endpoints. For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes.                                                                  |
+| `BIND`                         | `:8923`                 | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock`                                                                                                                                                                                                              |
+| `BIND_NETWORK`                 | `tcp`                   | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports.                                                                                                                                                                           |
+| `COOKIE_DOMAIN`                | unset                   | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See [here](https://stackoverflow.com/a/1063760) for more information.                                           |
+| `COOKIE_PARTITIONED`           | `false`                 | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe.                                                                                       |
+| `DIFFICULTY`                   | `4`                     | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses.                                                                                                                                                                                                               |
+| `ED25519_PRIVATE_KEY_HEX`      | unset                   | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details.                                                                                                                           |
+| `ED25519_PRIVATE_KEY_HEX_FILE` | unset                   | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set.                                                                                                                                                                                                     |
+| `METRICS_BIND`                 | `:9090`                 | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information.                                                                                                                                                                                                                       |
+| `METRICS_BIND_NETWORK`         | `tcp`                   | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information.                                                                                                                                                                                                               |
+| `OG_EXPIRY_TIME`               | `24h`                   | The expiration time for the Open Graph tag cache.                                                                                                                                                                                                                                                                    |
+| `OG_PASSTHROUGH`               | `false`                 | If set to `true`, Anubis will enable Open Graph tag passthrough.                                                                                                                                                                                                                                                     |
+| `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.mdx). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                                           |
+| `REDIRECT_DOMAINS`             | unset                   | If set, restrict the domains that Anubis can redirect to when passing a challenge.<br/><br/>If this is unset, Anubis may redirect to any domain which could cause security issues in the unlikely case that an attacker passes a challenge for your browser and then tricks you into clicking a link to your domain. |
+| `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file.                             |
+| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                                            |
+| `TARGET`                       | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`.                                                                                                                                                         |
+| `USE_REMOTE_ADDRESS`           | unset                   | If set to `true`, Anubis will take the client's IP from the network socket. For production deployments, it is expected that a reverse proxy is used in front of Anubis, which pass the IP using headers, instead.                                                                                                    |
+| `WEBMASTER_EMAIL`              | unset                   | If set, shows a contact email address when rendering error pages. This email address will be how users can get in contact with administrators.                                                                                                                                                                       |
 
 For more detailed information on configuring Open Graph tags, please refer to the [Open Graph Configuration](./configuration/open-graph.mdx) page.
 
+### Using Base Prefix
+
+The `BASE_PREFIX` environment variable allows you to run Anubis behind a path prefix. This is useful when:
+
+- You want to host multiple services on the same domain
+- You're using a reverse proxy that routes based on path prefixes
+- You need to integrate Anubis with an existing application structure
+
+For example, if you set `BASE_PREFIX=/myapp`, Anubis will:
+
+- Serve its challenge page at `/myapp/` instead of `/`
+- Serve its API endpoints at `/myapp/.within.website/x/cmd/anubis/api/` instead of `/.within.website/x/cmd/anubis/api/`
+- Serve its static assets at `/myapp/.within.website/x/cmd/anubis/` instead of `/.within.website/x/cmd/anubis/`
+
+When using this feature with a reverse proxy:
+
+1. Configure your reverse proxy to route requests for the specified path prefix to Anubis
+2. Set the `BASE_PREFIX` environment variable to match the path prefix in your reverse proxy configuration
+3. Ensure that your reverse proxy preserves the path when forwarding requests to Anubis
+
+Example with Nginx:
+
+```nginx
+location /myapp/ {
+    proxy_pass http://anubis:8923/myapp;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+}
+```
+
+With corresponding Anubis configuration:
+
+```
+BASE_PREFIX=/myapp
+```
+
 ### Key generation
 
 To generate an ed25519 private key, you can use this command:
@@ -91,3 +129,10 @@ To get Anubis filtering your traffic, you need to make sure it's added to your H
 - [Docker compose](./environments/docker-compose.mdx)
 - [Kubernetes](./environments/kubernetes.mdx)
 - [Nginx](./environments/nginx.mdx)
+- [Traefik](./environments/traefik.mdx)
+
+:::note
+
+Anubis loads its assets from `/.within.website/x/xess/` and `/.within.website/x/cmd/anubis`. If you do not reverse proxy these in your server config, Anubis won't work.
+
+:::

docs/docs/admin/native-install.mdx

@@ -86,20 +86,20 @@ Once it's installed, make a copy of the default configuration file `/etc/anubis/
 sudo cp /etc/anubis/default.env /etc/anubis/gitea.env
 ```
 
-Copy the default bot policies file to `/etc/anubis/gitea.botPolicies.json`:
+Copy the default bot policies file to `/etc/anubis/gitea.botPolicies.yaml`:
 
 <Tabs>
 <TabItem value="debrpm" label="Debian or Red Hat" default>
 
 ```text
-sudo cp /usr/share/doc/anubis/botPolicies.json /etc/anubis/gitea.botPolicies.json
+sudo cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/gitea.botPolicies.yaml
 ```
 
 </TabItem>
 <TabItem value="tarball" label="Tarball">
 
 ```text
-sudo cp ./doc/botPolicies.json /etc/anubis/gitea.botPolicies.json
+sudo cp ./doc/botPolicies.yaml /etc/anubis/gitea.botPolicies.yaml
 ```
 
 </TabItem>
@@ -114,7 +114,7 @@ BIND_NETWORK=tcp
 DIFFICULTY=4
 METRICS_BIND=[::1]:8240
 METRICS_BIND_NETWORK=tcp
-POLICY_FNAME=/etc/anubis/gitea.botPolicies.json
+POLICY_FNAME=/etc/anubis/gitea.botPolicies.yaml
 TARGET=http://localhost:3000
 ```
 

docs/docs/admin/policies.mdx

@@ -2,15 +2,25 @@
 title: Policy Definitions
 ---
 
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
 Out of the box, Anubis is pretty heavy-handed. It will aggressively challenge everything that might be a browser (usually indicated by having `Mozilla` in its user agent). However, some bots are smart enough to get past the challenge. Some things that look like bots may actually be fine (IE: RSS readers). Some resources need to be visible no matter what. Some resources and remotes are fine to begin with.
 
 Bot policies let you customize the rules that Anubis uses to allow, deny, or challenge incoming requests. Currently you can set policies by the following matches:
 
 - Request path
 - User agent string
+- HTTP request header values
+- [Importing other configuration snippets](./configuration/import.mdx)
+
+As of version v1.17.0 or later, configuration can be written in either JSON or YAML.
 
 Here's an example rule that denies [Amazonbot](https://developer.amazon.com/en/amazonbot):
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "amazonbot",
@@ -19,15 +29,37 @@ Here's an example rule that denies [Amazonbot](https://developer.amazon.com/en/a
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: amazonbot
+  user_agent_regex: Amazonbot
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
 When this rule is evaluated, Anubis will check the `User-Agent` string of the request. If it contains `Amazonbot`, Anubis will send an error page to the user saying that access is denied, but in such a way that makes scrapers think they have correctly loaded the webpage.
 
 Right now the only kinds of policies you can write are bot policies. Other forms of policies will be added in the future.
 
 Here is a minimal policy file that will protect against most scraper bots:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "bots": [
+    {
+      "name": "cloudflare-workers",
+      "headers_regex": {
+        "CF-Worker": ".*"
+      },
+      "action": "DENY"
+    },
     {
       "name": "well-known",
       "path_regex": "^/.well-known/.*$",
@@ -52,9 +84,35 @@ Here is a minimal policy file that will protect against most scraper bots:
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+bots:
+  - name: cloudflare-workers
+    headers_regex:
+      CF-Worker: .*
+    action: DENY
+  - name: well-known
+    path_regex: ^/.well-known/.*$
+    action: ALLOW
+  - name: favicon
+    path_regex: ^/favicon.ico$
+    action: ALLOW
+  - name: robots-txt
+    path_regex: ^/robots.txt$
+    action: ALLOW
+  - name: generic-browser
+    user_agent_regex: Mozilla
+    action: CHALLENGE
+```
+
+</TabItem>
+</Tabs>
+
 This allows requests to [`/.well-known`](https://en.wikipedia.org/wiki/Well-known_URI), `/favicon.ico`, `/robots.txt`, and challenges any request that has the word `Mozilla` in its User-Agent string. The [default policy file](https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.json) is a bit more cohesive, but this should be more than enough for most users.
 
-If no rules match the request, it is allowed through.
+If no rules match the request, it is allowed through. For more details on this default behavior and its implications, see [Default allow behavior](./default-allow-behavior.mdx).
 
 ## Writing your own rules
 
@@ -72,6 +130,11 @@ Name your rules in lower case using kebab-case. Rule names will be exposed in Pr
 
 Rules can also have their own challenge settings. These are customized using the `"challenge"` key. For example, here is a rule that makes challenges artificially hard for connections with the substring "bot" in their user agent:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+This rule has been known to have a high false positive rate in testing. Please use this with care.
+
 ```json
 {
   "name": "generic-bot-catchall",
@@ -85,6 +148,25 @@ Rules can also have their own challenge settings. These are customized using the
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+This rule has been known to have a high false positive rate in testing. Please use this with care.
+
+```yaml
+# Punish any bot with "bot" in the user-agent string
+- name: generic-bot-catchall
+  user_agent_regex: (?i:bot|crawler)
+  action: CHALLENGE
+  challenge:
+    difficulty: 16 # impossible
+    report_as: 4 # lie to the operator
+    algorithm: slow # intentionally waste CPU cycles and time
+```
+
+</TabItem>
+</Tabs>
+
 Challenges can be configured with these settings:
 
 | Key          | Example  | Description                                                                                                                                                                                    |
@@ -99,6 +181,9 @@ The `remote_addresses` field of a Bot rule allows you to set the IP range that t
 
 For example, you can allow a search engine to connect if and only if its IP address matches the ones they published:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "qwantbot",
@@ -108,8 +193,25 @@ For example, you can allow a search engine to connect if and only if its IP addr
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: qwantbot
+  user_agent_regex: \+https\://help\.qwant\.com/bot/
+  action: ALLOW
+  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
+  remote_addresses: ["91.242.162.0/24"]
+```
+
+</TabItem>
+</Tabs>
+
 This also works at an IP range level without any other checks:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "internal-network",
@@ -118,6 +220,19 @@ This also works at an IP range level without any other checks:
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+name: internal-network
+action: ALLOW
+remote_addresses:
+  - 100.64.0.0/10
+```
+
+</TabItem>
+</Tabs>
+
 ## Risk calculation for downstream services
 
 In case your service needs it for risk calculation reasons, Anubis exposes information about the rules that any requests match using a few headers:
@@ -126,6 +241,6 @@ In case your service needs it for risk calculation reasons, Anubis exposes infor
 | :---------------- | :--------------------------------------------------- | :--------------- |
 | `X-Anubis-Rule`   | The name of the rule that was matched                | `bot/lightpanda` |
 | `X-Anubis-Action` | The action that Anubis took in response to that rule | `CHALLENGE`      |
-| `X-Anubis-Status` | The status and how strict Anubis was in its checks   | `PASS-FULL`      |
+| `X-Anubis-Status` | The status and how strict Anubis was in its checks   | `PASS`           |
 
 Policy rules are matched using [Go's standard library regular expressions package](https://pkg.go.dev/regexp). You can mess around with the syntax at [regex101.com](https://regex101.com), make sure to select the Golang option.

docs/docs/funding.md

@@ -7,4 +7,4 @@ Anubis is provided to the public for free in order to help advance the common go
 
 If you want to run an unbranded or white-label version of Anubis, please [contact Xe](https://xeiaso.net/contact) to arrange a contract. This is not meant to be "contact us" pricing, I am still evaluating the market for this solution and figuring out what makes sense.
 
-You can donate to the project [on Patreon](https://patreon.com/cadey).
+You can donate to the project [on Patreon](https://patreon.com/cadey) or via [GitHub Sponsors](https://github.com/sponsors/Xe).

docs/docs/index.mdx

@@ -15,25 +15,50 @@ title: Anubis
 ![language count](https://img.shields.io/github/languages/count/TecharoHQ/anubis)
 ![repo size](https://img.shields.io/github/repo-size/TecharoHQ/anubis)
 
-Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.
+## Sponsors
+
+Anubis is brought to you by sponsors and donors like:
+
+[![Distrust](/img/sponsors/distrust-logo.webp)](https://distrust.co)
+
+## Overview
+
+Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a proof-of-work challenge in order to protect upstream resources from scraper bots.
 
 This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
 
-Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](./admin/policies.md) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
+Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](https://anubis.techaro.lol/docs/admin/policies) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
+
+In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.
 
 ## Support
 
 If you run into any issues running Anubis, please [open an issue](https://github.com/TecharoHQ/anubis/issues/new?template=Blank+issue) and include all the information I would need to diagnose your issue.
 
-For live chat, please join the [Patreon](https://patreon.com/cadey) and ask in the Patron discord in the channel `#anubis`.
+For live chat, please join the [Patreon](https://patreon.com/cadey) or join [GitHub Sponsors](https://github.com/sponsors/Xe) and ask in the Patron discord in the channel `#anubis`.
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date)](https://www.star-history.com/#TecharoHQ/anubis&Date)
+<a href="https://www.star-history.com/#TecharoHQ/anubis&Date">
+  <picture>
+    <source
+      media="(prefers-color-scheme: dark)"
+      srcSet="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date&theme=dark"
+    />
+    <source
+      media="(prefers-color-scheme: light)"
+      srcSet="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date"
+    />
+    <img
+      alt="Star History Chart"
+      src="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date"
+    />
+  </picture>
+</a>
 
 ## Packaging Status
 
-[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg)](https://repology.org/project/anubis-anti-crawler/versions)
+[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg?columns=3)](https://repology.org/project/anubis-anti-crawler/versions)
 
 ## Contributors
 

docs/docs/user/known-instances.md

@@ -4,28 +4,34 @@ title: List of known websites using Anubis
 
 This page contains a non-exhaustive list with all websites using Anubis.
 
-* <details>
+- <details>
   <summary>The Linux Foundation</summary>
-  
-  * https://git.kernel.org/
-  * https://lore.kernel.org/
+  - https://git.kernel.org/
+  - https://lore.kernel.org/
   </details>
-  * https://gitlab.gnome.org/
-  * https://scioly.org/
-  * https://bugs.winehq.org/
-  * https://svnweb.freebsd.org/
-  * https://trac.ffmpeg.org/
-  * https://git.sr.ht/
-  * https://xeiaso.net/
-  * https://source.puri.sm/
-  * https://git.enlightenment.org/
-  * https://superlove.sayitditto.net/
-  * https://linktaco.com/
-  * https://jaredallard.dev/
-  * https://dev.sanctum.geek.nz/
-  * https://canine.tools/
-* <details>
+- https://gitlab.gnome.org/
+- https://scioly.org/
+- https://bugs.winehq.org/
+- https://svnweb.freebsd.org/
+- https://trac.ffmpeg.org/
+- https://git.sr.ht/
+- https://xeiaso.net/
+- https://source.puri.sm/
+- https://git.enlightenment.org/
+- https://superlove.sayitditto.net/
+- https://linktaco.com/
+- https://jaredallard.dev/
+- https://dev.sanctum.geek.nz/
+- https://canine.tools/
+- https://git.lupancham.net/
+- https://dev.haiku-os.org
+- http://code.hackerspace.pl/
+- https://wiki.archlinux.org/
+- https://git.devuan.org/
+- https://hydra.nixos.org/
+
+- <details>
   <summary>The United Nations</summary>
-  
-  * https://policytoolbox.iiep.unesco.org/
-  </details>
+
+  - https://policytoolbox.iiep.unesco.org/
+  </details>

docs/docusaurus.config.ts

@@ -70,6 +70,9 @@ const config: Config = {
   ],
 
   themeConfig: {
+    colorMode: {
+      respectPrefersColorScheme: true,
+    },
     // Replace with your project's social card
     image: 'img/docusaurus-social-card.jpg',
     navbar: {

docs/package-lock.json

@@ -8512,9 +8512,9 @@
       }
     },
     "node_modules/estree-util-value-to-estree": {
-      "version": "3.3.2",
-      "resolved": "https://registry.npmjs.org/estree-util-value-to-estree/-/estree-util-value-to-estree-3.3.2.tgz",
-      "integrity": "sha512-hYH1aSvQI63Cvq3T3loaem6LW4u72F187zW4FHpTrReJSm6W66vYTFNO1vH/chmcOulp1HlAj1pxn8Ag0oXI5Q==",
+      "version": "3.3.3",
+      "resolved": "https://registry.npmjs.org/estree-util-value-to-estree/-/estree-util-value-to-estree-3.3.3.tgz",
+      "integrity": "sha512-Db+m1WSD4+mUO7UgMeKkAwdbfNWwIxLt48XF2oFU9emPfXkIu+k5/nlOj313v7wqtAPo0f9REhUvznFrPkG8CQ==",
       "license": "MIT",
       "dependencies": {
         "@types/estree": "^1.0.0"
@@ -10093,9 +10093,9 @@
       }
     },
     "node_modules/http-proxy-middleware": {
-      "version": "2.0.7",
-      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.7.tgz",
-      "integrity": "sha512-fgVY8AV7qU7z/MmXJ/rxwbrtQH4jBQ9m7kp3llF0liB7glmFeVZFBepQb32T3y8n8k2+AEYuMPCpinYW+/CuRA==",
+      "version": "2.0.9",
+      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.9.tgz",
+      "integrity": "sha512-c1IyJYLYppU574+YI7R4QyX2ystMtVXZwIdzazUIPIJsHuWNd+mho2j+bKoHftndicGj9yh+xjd+l0yj7VeT1Q==",
       "license": "MIT",
       "dependencies": {
         "@types/http-proxy": "^1.17.8",

docs/src/css/custom.css

@@ -21,16 +21,17 @@
 
 /* For readability concerns, you should choose a lighter palette in dark mode. */
 [data-theme="dark"] {
-  --ifm-color-primary: #25c2a0;
-  --ifm-color-primary-dark: #21af90;
-  --ifm-color-primary-darker: #1fa588;
-  --ifm-color-primary-darkest: #1a8870;
-  --ifm-color-primary-light: #29d5b0;
-  --ifm-color-primary-lighter: #32d8b4;
-  --ifm-color-primary-lightest: #4fddbf;
-  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
-  --code-block-diff-add-line-color: #216932;
-  --code-block-diff-remove-line-color: #8b423b;
+  --ifm-color-primary: #e64a19;
+  --ifm-color-primary-dark: #b73a12;
+  --ifm-color-primary-darker: #8c2c0e;
+  --ifm-color-primary-darkest: #5a1e0a;
+  --ifm-color-primary-light: #eb6d45;
+  --ifm-color-primary-lighter: #f09178;
+  --ifm-color-primary-lightest: #f5b5a6;
+  --ifm-code-font-size: 95%;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.25);
+  --code-block-diff-add-line-color: #2d5a2c;
+  --code-block-diff-remove-line-color: #5a2d2c;
 }
 
 .code-block-diff-add-line {

go.mod

@@ -45,6 +45,9 @@ require (
 	golang.org/x/tools v0.31.0 // indirect
 	google.golang.org/protobuf v1.36.5 // indirect
 	honnef.co/go/tools v0.6.1 // indirect
+	k8s.io/apimachinery v0.32.3 // indirect
+	sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect
+	sigs.k8s.io/yaml v1.4.0 // indirect
 )
 
 tool (

go.sum

@@ -138,3 +138,9 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.6.1 h1:R094WgE8K4JirYjBaOpz/AvTyUu/3wbmAoskKN/pxTI=
 honnef.co/go/tools v0.6.1/go.mod h1:3puzxxljPCe8RGJX7BIy1plGbxEOZni5mR2aXe3/uk4=
+k8s.io/apimachinery v0.32.3 h1:JmDuDarhDmA/Li7j3aPrwhpNBA94Nvk5zLeOge9HH1U=
+k8s.io/apimachinery v0.32.3/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 h1:/Rv+M11QRah1itp8VhT6HoVx1Ray9eB4DBr+K+/sCJ8=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3/go.mod h1:18nIHnGi6636UCz6m8i4DhaJ65T6EruyzmoQqI2BVDo=
+sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
+sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=

internal/headers.go

@@ -1,15 +1,29 @@
 package internal
 
 import (
+	"errors"
+	"fmt"
 	"log/slog"
 	"net"
 	"net/http"
+	"net/netip"
 	"strings"
 
 	"github.com/TecharoHQ/anubis"
 	"github.com/sebest/xff"
 )
 
+// TODO: move into config
+type XFFComputePreferences struct {
+	StripPrivate  bool
+	StripLoopback bool
+	StripCGNAT    bool
+	StripLLU      bool
+	Flatten       bool
+}
+
+var CGNat = netip.MustParsePrefix("100.64.0.0/10")
+
 // UnchangingCache sets the Cache-Control header to cache a response for 1 year if
 // and only if the application is compiled in "release" mode by Docker.
 func UnchangingCache(next http.Handler) http.Handler {
@@ -65,6 +79,112 @@ func XForwardedForToXRealIP(next http.Handler) http.Handler {
 	})
 }
 
+// XForwardedForUpdate sets or updates the X-Forwarded-For header, adding
+// the known remote address to an existing chain if present
+func XForwardedForUpdate(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		defer next.ServeHTTP(w, r)
+
+		pref := XFFComputePreferences{
+			StripPrivate:  true,
+			StripLoopback: true,
+			StripCGNAT:    true,
+			Flatten:       true,
+			StripLLU:      true,
+		}
+
+		remoteAddr := r.RemoteAddr
+		origXFFHeader := r.Header.Get("X-Forwarded-For")
+
+		if remoteAddr == "@" {
+			// remote is a unix socket
+			// do not touch chain
+			return
+		}
+
+		xffHeaderString, err := computeXFFHeader(remoteAddr, origXFFHeader, pref)
+		if err != nil {
+			slog.Debug("computing X-Forwarded-For header failed", "err", err)
+			return
+		}
+
+		if len(xffHeaderString) == 0 {
+			r.Header.Del("X-Forwarded-For")
+		} else {
+			r.Header.Set("X-Forwarded-For", xffHeaderString)
+		}
+	})
+}
+
+var (
+	ErrCantSplitHostParse = errors.New("internal: unable to net.SplitHostParse")
+	ErrCantParseRemoteIP  = errors.New("internal: unable to parse remote IP")
+)
+
+func computeXFFHeader(remoteAddr string, origXFFHeader string, pref XFFComputePreferences) (string, error) {
+	remoteIP, _, err := net.SplitHostPort(remoteAddr)
+	if err != nil {
+		return "", fmt.Errorf("%w: %w", ErrCantSplitHostParse, err)
+	}
+	parsedRemoteIP, err := netip.ParseAddr(remoteIP)
+	if err != nil {
+		return "", fmt.Errorf("%w: %w", ErrCantParseRemoteIP, err)
+	}
+
+	origForwardedList := make([]string, 0, 4)
+	if origXFFHeader != "" {
+		origForwardedList = strings.Split(origXFFHeader, ",")
+	}
+	origForwardedList = append(origForwardedList, parsedRemoteIP.String())
+	forwardedList := make([]string, 0, len(origForwardedList))
+	// this behavior is equivalent to
+	// ingress-nginx "compute-full-forwarded-for"
+	// https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#compute-full-forwarded-for
+	//
+	// this would be the correct place to strip and/or flatten this list
+	//
+	// strip - iterate backwards and eliminate configured trusted IPs
+	// flatten - only return the last element to avoid spoofing confusion
+	//
+	// many applications handle this in different ways, but
+	// generally they'd be expected to do these two things on
+	// their own end to find the first non-spoofed IP
+	for i := len(origForwardedList) - 1; i >= 0; i-- {
+		segmentIP, err := netip.ParseAddr(origForwardedList[i])
+		if err != nil {
+			// can't assess this element, so the remainder of the chain
+			// can't be trusted. not a fatal error, since anyone can
+			// spoof an XFF header
+			slog.Debug("failed to parse XFF segment", "err", err)
+			break
+		}
+		if pref.StripPrivate && segmentIP.IsPrivate() {
+			continue
+		}
+		if pref.StripLoopback && segmentIP.IsLoopback() {
+			continue
+		}
+		if pref.StripLLU && segmentIP.IsLinkLocalUnicast() {
+			continue
+		}
+		if pref.StripCGNAT && CGNat.Contains(segmentIP) {
+			continue
+		}
+		forwardedList = append([]string{segmentIP.String()}, forwardedList...)
+	}
+	var xffHeaderString string
+	if len(forwardedList) == 0 {
+		xffHeaderString = ""
+		return xffHeaderString, nil
+	}
+	if pref.Flatten {
+		xffHeaderString = forwardedList[len(forwardedList)-1]
+	} else {
+		xffHeaderString = strings.Join(forwardedList, ",")
+	}
+	return xffHeaderString, nil
+}
+
 // NoStoreCache sets the Cache-Control header to no-store for the response.
 func NoStoreCache(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -73,7 +193,7 @@ func NoStoreCache(next http.Handler) http.Handler {
 	})
 }
 
-// Do not allow browsing directory listings in paths that end with /
+// NoBrowsing prevents directory browsing by returning a 404 for any request that ends with a "/".
 func NoBrowsing(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		if strings.HasSuffix(r.URL.Path, "/") {

internal/ogtags/fetch.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"fmt"
 	"golang.org/x/net/html"
+	"io"
 	"log/slog"
 	"mime"
 	"net"
@@ -26,7 +27,12 @@ func (c *OGTagCache) fetchHTMLDocument(urlStr string) (*html.Node, error) {
 		return nil, fmt.Errorf("http get failed: %w", err)
 	}
 	// this defer will call MaxBytesReader's Close, which closes the original body.
-	defer resp.Body.Close()
+	defer func(Body io.ReadCloser) {
+		err := Body.Close()
+		if err != nil {
+			slog.Debug("og: error closing response body", "url", urlStr, "error", err)
+		}
+	}(resp.Body)
 
 	if resp.StatusCode != http.StatusOK {
 		slog.Debug("og: received non-OK status code", "url", urlStr, "status", resp.StatusCode)

internal/test/playwright_test.go

@@ -265,6 +265,132 @@ func TestPlaywrightBrowser(t *testing.T) {
 	}
 }
 
+func TestPlaywrightWithBasePrefix(t *testing.T) {
+	if os.Getenv("DONT_USE_NETWORK") != "" {
+		t.Skip("test requires network egress")
+		return
+	}
+
+	t.Skip("NOTE(Xe)\\ these tests require HTTPS support in #364")
+
+	doesNPXExist(t)
+	startPlaywright(t)
+
+	pw := setupPlaywright(t)
+	basePrefix := "/myapp"
+	anubisURL := spawnAnubisWithOptions(t, basePrefix)
+
+	// Reset BasePrefix after test
+	t.Cleanup(func() {
+		anubis.BasePrefix = ""
+	})
+
+	browsers := []playwright.BrowserType{pw.Chromium}
+
+	for _, typ := range browsers {
+		t.Run(typ.Name()+"/basePrefix", func(t *testing.T) {
+			browser, err := typ.Connect(buildBrowserConnect(typ.Name()), playwright.BrowserTypeConnectOptions{
+				ExposeNetwork: playwright.String("<loopback>"),
+			})
+			if err != nil {
+				t.Fatalf("could not connect to remote browser: %v", err)
+			}
+			defer browser.Close()
+
+			ctx, err := browser.NewContext(playwright.BrowserNewContextOptions{
+				AcceptDownloads: playwright.Bool(false),
+				ExtraHttpHeaders: map[string]string{
+					"X-Real-Ip": "127.0.0.1",
+				},
+				UserAgent: playwright.String("Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"),
+			})
+			if err != nil {
+				t.Fatalf("could not create context: %v", err)
+			}
+			defer ctx.Close()
+
+			page, err := ctx.NewPage()
+			if err != nil {
+				t.Fatalf("could not create page: %v", err)
+			}
+			defer page.Close()
+
+			// Test accessing the base URL with prefix
+			_, err = page.Goto(anubisURL+basePrefix, playwright.PageGotoOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(5*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not navigate to test server with base prefix: %v", err)
+			}
+
+			// Check if challenge page is displayed
+			image := page.Locator("#image[src*=pensive], #image[src*=happy]")
+			err = image.WaitFor(playwright.LocatorWaitForOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(5*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not wait for challenge image: %v", err)
+			}
+
+			isVisible, err := image.IsVisible()
+			if err != nil {
+				pwFail(t, page, "could not check if challenge image is visible: %v", err)
+			}
+			if !isVisible {
+				pwFail(t, page, "challenge image not visible")
+			}
+
+			// Complete the challenge
+			// Wait for the challenge to be solved
+			anubisTest := page.Locator("#anubis-test")
+			err = anubisTest.WaitFor(playwright.LocatorWaitForOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(30*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not wait for challenge to be solved: %v", err)
+			}
+
+			// Verify the challenge was solved
+			content, err := anubisTest.TextContent(playwright.LocatorTextContentOptions{})
+			if err != nil {
+				pwFail(t, page, "could not get text content: %v", err)
+			}
+
+			var tm int64
+			if _, err := fmt.Sscanf(content, "%d", &tm); err != nil {
+				pwFail(t, page, "unexpected output: %s", content)
+			}
+
+			// Check if the timestamp is reasonable
+			now := time.Now().Unix()
+			if tm < now-60 || tm > now+60 {
+				pwFail(t, page, "unexpected timestamp in output: %d not in range %d±60", tm, now)
+			}
+
+			// Check if cookie has the correct path
+			cookies, err := ctx.Cookies()
+			if err != nil {
+				pwFail(t, page, "could not get cookies: %v", err)
+			}
+
+			var found bool
+			for _, cookie := range cookies {
+				if cookie.Name == anubis.CookieName {
+					found = true
+					if cookie.Path != basePrefix+"/" {
+						t.Errorf("cookie path is wrong, wanted %s, got: %s", basePrefix+"/", cookie.Path)
+					}
+					break
+				}
+			}
+
+			if !found {
+				t.Errorf("Cookie %q not found", anubis.CookieName)
+			}
+		})
+	}
+}
+
 func buildBrowserConnect(name string) string {
 	u, _ := url.Parse(*playwrightServer)
 
@@ -378,14 +504,14 @@ func pwFail(t *testing.T, page playwright.Page, format string, args ...any) erro
 }
 
 func pwTimeout(tc testCase, deadline time.Time) *float64 {
-	max := *playwrightMaxTime
+	maxTime := *playwrightMaxTime
 	if tc.isHard {
-		max = *playwrightMaxHardTime
+		maxTime = *playwrightMaxHardTime
 	}
 
 	d := time.Until(deadline)
-	if d <= 0 || d > max {
-		return playwright.Float(float64(max.Milliseconds()))
+	if d <= 0 || d > maxTime {
+		return playwright.Float(float64(maxTime.Milliseconds()))
 	}
 	return playwright.Float(float64(d.Milliseconds()))
 }
@@ -431,6 +557,10 @@ func setupPlaywright(t *testing.T) *playwright.Playwright {
 }
 
 func spawnAnubis(t *testing.T) string {
+	return spawnAnubisWithOptions(t, "")
+}
+
+func spawnAnubisWithOptions(t *testing.T, basePrefix string) string {
 	t.Helper()
 
 	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -457,6 +587,7 @@ func spawnAnubis(t *testing.T) string {
 		Policy:         policy,
 		ServeRobotsTXT: true,
 		Target:         "http://" + host + ":" + port,
+		BasePrefix:     basePrefix,
 	})
 	if err != nil {
 		t.Fatalf("can't construct libanubis.Server: %v", err)

internal/xff_test.go

@@ -0,0 +1,166 @@
+package internal
+
+import (
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestXForwardedForUpdateIgnoreUnix(t *testing.T) {
+	var remoteAddr = ""
+	var xff = ""
+
+	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		remoteAddr = r.RemoteAddr
+		xff = r.Header.Get("X-Forwarded-For")
+		w.WriteHeader(http.StatusOK)
+	})
+
+	r := httptest.NewRequest(http.MethodGet, "/", nil)
+
+	r.RemoteAddr = "@"
+
+	w := httptest.NewRecorder()
+
+	XForwardedForUpdate(h).ServeHTTP(w, r)
+
+	if r.RemoteAddr != remoteAddr {
+		t.Errorf("wanted remoteAddr to be %s, got: %s", r.RemoteAddr, remoteAddr)
+	}
+
+	if xff != "" {
+		t.Error("handler added X-Forwarded-For when it should not have")
+	}
+}
+
+func TestXForwardedForUpdateAddToChain(t *testing.T) {
+	var xff = ""
+	const expected = "1.1.1.1"
+
+	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		xff = r.Header.Get("X-Forwarded-For")
+		w.WriteHeader(http.StatusOK)
+	})
+
+	srv := httptest.NewServer(XForwardedForUpdate(h))
+
+	r, err := http.NewRequest(http.MethodGet, srv.URL, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	r.Header.Set("X-Forwarded-For", "1.1.1.1,10.20.30.40")
+
+	if _, err := srv.Client().Do(r); err != nil {
+		t.Fatal(err)
+	}
+
+	if xff != expected {
+		t.Logf("expected: %s", expected)
+		t.Logf("got:      %s", xff)
+		t.Error("X-Forwarded-For header was not what was expected")
+	}
+}
+
+func TestComputeXFFHeader(t *testing.T) {
+	for _, tt := range []struct {
+		name          string
+		remoteAddr    string
+		origXFFHeader string
+		pref          XFFComputePreferences
+		result        string
+		err           error
+	}{
+		{
+			name:          "StripPrivate",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1",
+			pref: XFFComputePreferences{
+				StripPrivate: true,
+			},
+			result: "1.1.1.1,127.0.0.1",
+		},
+		{
+			name:          "StripLoopback",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,127.0.0.1",
+			pref: XFFComputePreferences{
+				StripLoopback: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripCGNAT",
+			remoteAddr:    "100.64.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,100.64.0.1",
+			pref: XFFComputePreferences{
+				StripCGNAT: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripLinkLocalUnicastIPv4",
+			remoteAddr:    "169.254.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,169.254.0.1",
+			pref: XFFComputePreferences{
+				StripLLU: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripLinkLocalUnicastIPv6",
+			remoteAddr:    "169.254.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,fe80::",
+			pref: XFFComputePreferences{
+				StripLLU: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "Flatten",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,fe80::,100.64.0.1,169.254.0.1",
+			pref: XFFComputePreferences{
+				StripPrivate:  true,
+				StripLoopback: true,
+				StripCGNAT:    true,
+				StripLLU:      true,
+				Flatten:       true,
+			},
+			result: "1.1.1.1",
+		},
+		{
+			name:       "invalid-ip-port",
+			remoteAddr: "fe80::",
+			err:        ErrCantSplitHostParse,
+		},
+		{
+			name:       "invalid-remote-ip",
+			remoteAddr: "anubis:80",
+			err:        ErrCantParseRemoteIP,
+		},
+		{
+			name:       "no-xff-dont-panic",
+			remoteAddr: "127.0.0.1:80",
+			pref: XFFComputePreferences{
+				StripPrivate:  true,
+				StripLoopback: true,
+				StripCGNAT:    true,
+				StripLLU:      true,
+				Flatten:       true,
+			},
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			result, err := computeXFFHeader(tt.remoteAddr, tt.origXFFHeader, tt.pref)
+			if err != nil && !errors.Is(err, tt.err) {
+				t.Errorf("computeXFFHeader got the wrong error, wanted %v but got: %v", tt.err, err)
+			}
+
+			if result != tt.result {
+				t.Errorf("computeXFFHeader returned the wrong result, wanted %q but got: %q", tt.result, result)
+			}
+		})
+	}
+}

lib/anubis.go

@@ -8,12 +8,13 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"log"
 	"log/slog"
 	"math"
 	"net"
 	"net/http"
+	"net/url"
 	"os"
+	"slices"
 	"strconv"
 	"strings"
 	"time"
@@ -64,10 +65,11 @@ var (
 )
 
 type Options struct {
-	Next           http.Handler
-	Policy         *policy.ParsedConfig
-	ServeRobotsTXT bool
-	PrivateKey     ed25519.PrivateKey
+	Next            http.Handler
+	Policy          *policy.ParsedConfig
+	RedirectDomains []string
+	ServeRobotsTXT  bool
+	PrivateKey      ed25519.PrivateKey
 
 	CookieDomain      string
 	CookieName        string
@@ -78,6 +80,7 @@ type Options struct {
 	Target        string
 
 	WebmasterEmail string
+	BasePrefix     string
 }
 
 func LoadPoliciesOrDefault(fname string, defaultDifficulty int) (*policy.ParsedConfig, error) {
@@ -90,14 +93,19 @@ func LoadPoliciesOrDefault(fname string, defaultDifficulty int) (*policy.ParsedC
 			return nil, fmt.Errorf("can't parse policy file %s: %w", fname, err)
 		}
 	} else {
-		fname = "(data)/botPolicies.json"
-		fin, err = data.BotPolicies.Open("botPolicies.json")
+		fname = "(data)/botPolicies.yaml"
+		fin, err = data.BotPolicies.Open("botPolicies.yaml")
 		if err != nil {
 			return nil, fmt.Errorf("[unexpected] can't parse builtin policy file %s: %w", fname, err)
 		}
 	}
 
-	defer fin.Close()
+	defer func(fin io.ReadCloser) {
+		err := fin.Close()
+		if err != nil {
+			slog.Error("failed to close policy file", "file", fname, "err", err)
+		}
+	}(fin)
 
 	anubisPolicy, err := policy.ParseConfig(fin, fname, defaultDifficulty)
 
@@ -114,6 +122,8 @@ func New(opts Options) (*Server, error) {
 		opts.PrivateKey = priv
 	}
 
+	anubis.BasePrefix = opts.BasePrefix
+
 	result := &Server{
 		next:       opts.Next,
 		priv:       opts.PrivateKey,
@@ -127,25 +137,42 @@ func New(opts Options) (*Server, error) {
 	mux := http.NewServeMux()
 	xess.Mount(mux)
 
-	mux.Handle(anubis.StaticPath, internal.UnchangingCache(internal.NoBrowsing(http.StripPrefix(anubis.StaticPath, http.FileServerFS(web.Static)))))
+	// Helper to add global prefix
+	registerWithPrefix := func(pattern string, handler http.Handler, method string) {
+		if method != "" {
+			method = method + " " // methods must end with a space to register with them
+		}
 
-	if opts.ServeRobotsTXT {
-		mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, r *http.Request) {
-			http.ServeFileFS(w, r, web.Static, "static/robots.txt")
-		})
+		// Ensure there's no double slash when concatenating BasePrefix and pattern
+		basePrefix := strings.TrimSuffix(anubis.BasePrefix, "/")
+		prefix := method + basePrefix
 
-		mux.HandleFunc("/.well-known/robots.txt", func(w http.ResponseWriter, r *http.Request) {
-			http.ServeFileFS(w, r, web.Static, "static/robots.txt")
-		})
+		// If pattern doesn't start with a slash, add one
+		if !strings.HasPrefix(pattern, "/") {
+			pattern = "/" + pattern
+		}
+
+		mux.Handle(prefix+pattern, handler)
 	}
 
-	// mux.HandleFunc("GET /.within.website/x/cmd/anubis/static/js/main.mjs", serveMainJSWithBestEncoding)
+	// Ensure there's no double slash when concatenating BasePrefix and StaticPath
+	stripPrefix := strings.TrimSuffix(anubis.BasePrefix, "/") + anubis.StaticPath
+	registerWithPrefix(anubis.StaticPath, internal.UnchangingCache(internal.NoBrowsing(http.StripPrefix(stripPrefix, http.FileServerFS(web.Static)))), "")
 
-	mux.HandleFunc("POST /.within.website/x/cmd/anubis/api/make-challenge", result.MakeChallenge)
-	mux.HandleFunc("GET /.within.website/x/cmd/anubis/api/pass-challenge", result.PassChallenge)
-	mux.HandleFunc("GET /.within.website/x/cmd/anubis/api/test-error", result.TestError)
+	if opts.ServeRobotsTXT {
+		registerWithPrefix("/robots.txt", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			http.ServeFileFS(w, r, web.Static, "static/robots.txt")
+		}), "GET")
+		registerWithPrefix("/.well-known/robots.txt", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			http.ServeFileFS(w, r, web.Static, "static/robots.txt")
+		}), "GET")
+	}
 
-	mux.HandleFunc("/", result.MaybeReverseProxy)
+	registerWithPrefix(anubis.APIPrefix+"make-challenge", http.HandlerFunc(result.MakeChallenge), "POST")
+	registerWithPrefix(anubis.APIPrefix+"pass-challenge", http.HandlerFunc(result.PassChallenge), "GET")
+	registerWithPrefix(anubis.APIPrefix+"check", http.HandlerFunc(result.maybeReverseProxyHttpStatusOnly), "")
+	registerWithPrefix(anubis.APIPrefix+"test-error", http.HandlerFunc(result.TestError), "GET")
+	registerWithPrefix("/", http.HandlerFunc(result.maybeReverseProxyOrPage), "")
 
 	result.mux = mux
 
@@ -167,6 +194,36 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	s.mux.ServeHTTP(w, r)
 }
 
+func (s *Server) ServeHTTPNext(w http.ResponseWriter, r *http.Request) {
+	if s.next == nil {
+		redir := r.FormValue("redir")
+		urlParsed, err := r.URL.Parse(redir)
+		if err != nil {
+			templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect URL not parseable", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+			return
+		}
+
+		if len(urlParsed.Host) > 0 && len(s.opts.RedirectDomains) != 0 && !slices.Contains(s.opts.RedirectDomains, urlParsed.Host) {
+			templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect domain not allowed", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+			return
+		} else if urlParsed.Host != r.URL.Host {
+			templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect domain not allowed", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+			return
+		}
+
+		if redir != "" {
+			http.Redirect(w, r, redir, http.StatusFound)
+			return
+		}
+
+		templ.Handler(
+			web.Base("You are not a bot!", web.StaticHappy()),
+		).ServeHTTP(w, r)
+	} else {
+		s.next.ServeHTTP(w, r)
+	}
+}
+
 func (s *Server) challengeFor(r *http.Request, difficulty int) string {
 	fp := sha256.Sum256(s.priv.Seed())
 
@@ -182,7 +239,15 @@ func (s *Server) challengeFor(r *http.Request, difficulty int) string {
 	return internal.SHA256sum(challengeData)
 }
 
-func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
+func (s *Server) maybeReverseProxyHttpStatusOnly(w http.ResponseWriter, r *http.Request) {
+	s.maybeReverseProxy(w, r, true)
+}
+
+func (s *Server) maybeReverseProxyOrPage(w http.ResponseWriter, r *http.Request) {
+	s.maybeReverseProxy(w, r, false)
+}
+
+func (s *Server) maybeReverseProxy(w http.ResponseWriter, r *http.Request, httpStatusOnly bool) {
 	lg := slog.With(
 		"user_agent", r.UserAgent(),
 		"accept_language", r.Header.Get("Accept-Language"),
@@ -202,7 +267,7 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
 	r.Header.Add("X-Anubis-Rule", cr.Name)
 	r.Header.Add("X-Anubis-Action", string(cr.Rule))
 	lg = lg.With("check_result", cr)
-	policy.PolicyApplications.WithLabelValues(cr.Name, string(cr.Rule)).Add(1)
+	policy.Applications.WithLabelValues(cr.Name, string(cr.Rule)).Add(1)
 
 	ip := r.Header.Get("X-Real-Ip")
 
@@ -228,7 +293,7 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
 	switch cr.Rule {
 	case config.RuleAllow:
 		lg.Debug("allowing traffic to origin (explicit)")
-		s.next.ServeHTTP(w, r)
+		s.ServeHTTPNext(w, r)
 		return
 	case config.RuleDeny:
 		s.ClearCookie(w)
@@ -238,12 +303,8 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
 			templ.Handler(web.Base("Oh noes!", web.ErrorPage("Other internal server error (contact the admin)", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
 			return
 		}
-		hash, err := rule.Hash()
-		if err != nil {
-			lg.Error("can't calculate checksum of rule", "err", err)
-			templ.Handler(web.Base("Oh noes!", web.ErrorPage("Other internal server error (contact the admin)", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
-			return
-		}
+		hash := rule.Hash()
+
 		lg.Debug("rule hash", "hash", hash)
 		templ.Handler(web.Base("Oh noes!", web.ErrorPage(fmt.Sprintf("Access Denied: error code %s", hash), s.opts.WebmasterEmail)), templ.WithStatus(http.StatusOK)).ServeHTTP(w, r)
 		return
@@ -263,21 +324,21 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
 	if err != nil {
 		lg.Debug("cookie not found", "path", r.URL.Path)
 		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
+		s.RenderIndex(w, r, rule, httpStatusOnly)
 		return
 	}
 
 	if err := ckie.Valid(); err != nil {
 		lg.Debug("cookie is invalid", "err", err)
 		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
+		s.RenderIndex(w, r, rule, httpStatusOnly)
 		return
 	}
 
 	if time.Now().After(ckie.Expires) && !ckie.Expires.IsZero() {
 		lg.Debug("cookie expired", "path", r.URL.Path)
 		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
+		s.RenderIndex(w, r, rule, httpStatusOnly)
 		return
 	}
 
@@ -288,56 +349,21 @@ func (s *Server) MaybeReverseProxy(w http.ResponseWriter, r *http.Request) {
 	if err != nil || !token.Valid {
 		lg.Debug("invalid token", "path", r.URL.Path, "err", err)
 		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
+		s.RenderIndex(w, r, rule, httpStatusOnly)
 		return
 	}
 
-	if randomJitter() {
-		r.Header.Add("X-Anubis-Status", "PASS-BRIEF")
-		lg.Debug("cookie is not enrolled into secondary screening")
-		s.next.ServeHTTP(w, r)
-		return
-	}
-
-	claims, ok := token.Claims.(jwt.MapClaims)
-	if !ok {
-		lg.Debug("invalid token claims type", "path", r.URL.Path)
-		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
-		return
-	}
-	challenge := s.challengeFor(r, rule.Challenge.Difficulty)
-
-	if claims["challenge"] != challenge {
-		lg.Debug("invalid challenge", "path", r.URL.Path)
-		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
-		return
-	}
-
-	var nonce int
-
-	if v, ok := claims["nonce"].(float64); ok {
-		nonce = int(v)
-	}
-
-	calcString := fmt.Sprintf("%s%d", challenge, nonce)
-	calculated := internal.SHA256sum(calcString)
-
-	if subtle.ConstantTimeCompare([]byte(claims["response"].(string)), []byte(calculated)) != 1 {
-		lg.Debug("invalid response", "path", r.URL.Path)
-		failedValidations.Inc()
-		s.ClearCookie(w)
-		s.RenderIndex(w, r, cr, rule)
-		return
-	}
-
-	slog.Debug("all checks passed")
-	r.Header.Add("X-Anubis-Status", "PASS-FULL")
-	s.next.ServeHTTP(w, r)
+	r.Header.Add("X-Anubis-Status", "PASS")
+	s.ServeHTTPNext(w, r)
 }
 
-func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, cr CheckResult, rule *policy.Bot) {
+func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, rule *policy.Bot, returnHTTPStatusOnly bool) {
+	if returnHTTPStatusOnly {
+		w.WriteHeader(http.StatusUnauthorized)
+		w.Write([]byte("Authorization required"))
+		return
+	}
+
 	lg := slog.With(
 		"user_agent", r.UserAgent(),
 		"accept_language", r.Header.Get("Accept-Language"),
@@ -379,27 +405,37 @@ func (s *Server) RenderBench(w http.ResponseWriter, r *http.Request) {
 func (s *Server) MakeChallenge(w http.ResponseWriter, r *http.Request) {
 	lg := slog.With("user_agent", r.UserAgent(), "accept_language", r.Header.Get("Accept-Language"), "priority", r.Header.Get("Priority"), "x-forwarded-for", r.Header.Get("X-Forwarded-For"), "x-real-ip", r.Header.Get("X-Real-Ip"))
 
+	encoder := json.NewEncoder(w)
 	cr, rule, err := s.check(r)
 	if err != nil {
 		lg.Error("check failed", "err", err)
 		w.WriteHeader(http.StatusInternalServerError)
-		json.NewEncoder(w).Encode(struct {
+		err := encoder.Encode(struct {
 			Error string `json:"error"`
 		}{
 			Error: "Internal Server Error: administrator has misconfigured Anubis. Please contact the administrator and ask them to look for the logs around \"makeChallenge\"",
 		})
+		if err != nil {
+			lg.Error("failed to encode error response", "err", err)
+			w.WriteHeader(http.StatusInternalServerError)
+		}
 		return
 	}
 	lg = lg.With("check_result", cr)
 	challenge := s.challengeFor(r, rule.Challenge.Difficulty)
 
-	json.NewEncoder(w).Encode(struct {
+	err = encoder.Encode(struct {
 		Challenge string                 `json:"challenge"`
 		Rules     *config.ChallengeRules `json:"rules"`
 	}{
 		Challenge: challenge,
 		Rules:     rule.Challenge,
 	})
+	if err != nil {
+		lg.Error("failed to encode challenge", "err", err)
+		w.WriteHeader(http.StatusInternalServerError)
+		return
+	}
 	lg.Debug("made challenge", "challenge", challenge, "rules", rule.Challenge, "cr", cr)
 	challengesIssued.Inc()
 }
@@ -413,6 +449,16 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
 		"x-real-ip", r.Header.Get("X-Real-Ip"),
 	)
 
+	redir := r.FormValue("redir")
+	redirURL, err := url.ParseRequestURI(redir)
+	if err != nil {
+		lg.Error("invalid redirect", "err", err)
+		templ.Handler(web.Base("Oh noes!", web.ErrorPage("invalid redirect", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+		return
+	}
+	// used by the path checker rule
+	r.URL = redirURL
+
 	cr, rule, err := s.check(r)
 	if err != nil {
 		lg.Error("check failed", "err", err)
@@ -449,7 +495,19 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
 	timeTaken.Observe(elapsedTime)
 
 	response := r.FormValue("response")
-	redir := r.FormValue("redir")
+	urlParsed, err := r.URL.Parse(redir)
+	if err != nil {
+		templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect URL not parseable", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+		return
+	}
+
+	if len(urlParsed.Host) > 0 && len(s.opts.RedirectDomains) != 0 && !slices.Contains(s.opts.RedirectDomains, urlParsed.Host) {
+		templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect domain not allowed", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+		return
+	} else if urlParsed.Host != r.URL.Host {
+		templ.Handler(web.Base("Oh noes!", web.ErrorPage("Redirect domain not allowed", s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
+		return
+	}
 
 	challenge := s.challengeFor(r, rule.Challenge.Difficulty)
 
@@ -481,6 +539,11 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
+	// Adjust cookie path if base prefix is not empty
+	cookiePath := "/"
+	if anubis.BasePrefix != "" {
+		cookiePath = strings.TrimSuffix(anubis.BasePrefix, "/") + "/"
+	}
 	// generate JWT cookie
 	token := jwt.NewWithClaims(jwt.SigningMethodEdDSA, jwt.MapClaims{
 		"challenge": challenge,
@@ -505,7 +568,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
 		SameSite:    http.SameSiteLaxMode,
 		Domain:      s.opts.CookieDomain,
 		Partitioned: s.opts.CookiePartitioned,
-		Path:        "/",
+		Path:        cookiePath,
 	})
 
 	challengesValidated.Inc()
@@ -518,35 +581,33 @@ func (s *Server) TestError(w http.ResponseWriter, r *http.Request) {
 	templ.Handler(web.Base("Oh noes!", web.ErrorPage(err, s.opts.WebmasterEmail)), templ.WithStatus(http.StatusInternalServerError)).ServeHTTP(w, r)
 }
 
+func cr(name string, rule config.Rule) policy.CheckResult {
+	return policy.CheckResult{
+		Name: name,
+		Rule: rule,
+	}
+}
+
 // Check evaluates the list of rules, and returns the result
-func (s *Server) check(r *http.Request) (CheckResult, *policy.Bot, error) {
+func (s *Server) check(r *http.Request) (policy.CheckResult, *policy.Bot, error) {
 	host := r.Header.Get("X-Real-Ip")
 	if host == "" {
-		return decaymap.Zilch[CheckResult](), nil, fmt.Errorf("[misconfiguration] X-Real-Ip header is not set")
+		return decaymap.Zilch[policy.CheckResult](), nil, fmt.Errorf("[misconfiguration] X-Real-Ip header is not set")
 	}
 
 	addr := net.ParseIP(host)
 	if addr == nil {
-		return decaymap.Zilch[CheckResult](), nil, fmt.Errorf("[misconfiguration] %q is not an IP address", host)
+		return decaymap.Zilch[policy.CheckResult](), nil, fmt.Errorf("[misconfiguration] %q is not an IP address", host)
 	}
 
 	for _, b := range s.policy.Bots {
-		if b.UserAgent != nil {
-			if b.UserAgent.MatchString(r.UserAgent()) && s.checkRemoteAddress(b, addr) {
-				return cr("bot/"+b.Name, b.Action), &b, nil
-			}
+		match, err := b.Rules.Check(r)
+		if err != nil {
+			return decaymap.Zilch[policy.CheckResult](), nil, fmt.Errorf("can't run check %s: %w", b.Name, err)
 		}
 
-		if b.Path != nil {
-			if b.Path.MatchString(r.URL.Path) && s.checkRemoteAddress(b, addr) {
-				return cr("bot/"+b.Name, b.Action), &b, nil
-			}
-		}
-
-		if b.Ranger != nil {
-			if s.checkRemoteAddress(b, addr) {
-				return cr("bot/"+b.Name, b.Action), &b, nil
-			}
+		if match {
+			return cr("bot/"+b.Name, b.Action), &b, nil
 		}
 	}
 
@@ -559,19 +620,6 @@ func (s *Server) check(r *http.Request) (CheckResult, *policy.Bot, error) {
 	}, nil
 }
 
-func (s *Server) checkRemoteAddress(b policy.Bot, addr net.IP) bool {
-	if b.Ranger == nil {
-		return true
-	}
-
-	ok, err := b.Ranger.Contains(addr)
-	if err != nil {
-		log.Panicf("[unexpected] something very funky is going on, %q does not have a calculable network number: %v", addr.String(), err)
-	}
-
-	return ok
-}
-
 func (s *Server) CleanupDecayMap() {
 	s.DNSBLCache.Cleanup()
 	s.OGTags.Cleanup()

lib/anubis_test.go

@@ -5,9 +5,12 @@ import (
 	"fmt"
 	"net/http"
 	"net/http/httptest"
+	"os"
+	"strings"
 	"testing"
 
 	"github.com/TecharoHQ/anubis"
+	"github.com/TecharoHQ/anubis/data"
 	"github.com/TecharoHQ/anubis/internal"
 	"github.com/TecharoHQ/anubis/lib/policy"
 )
@@ -15,7 +18,7 @@ import (
 func loadPolicies(t *testing.T, fname string) *policy.ParsedConfig {
 	t.Helper()
 
-	anubisPolicy, err := LoadPoliciesOrDefault("", anubis.DefaultDifficulty)
+	anubisPolicy, err := LoadPoliciesOrDefault(fname, anubis.DefaultDifficulty)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -55,6 +58,22 @@ func makeChallenge(t *testing.T, ts *httptest.Server) challenge {
 	return chall
 }
 
+func TestLoadPolicies(t *testing.T) {
+	for _, fname := range []string{"botPolicies.json", "botPolicies.yaml"} {
+		t.Run(fname, func(t *testing.T) {
+			fin, err := data.BotPolicies.Open(fname)
+			if err != nil {
+				t.Fatal(err)
+			}
+			defer fin.Close()
+
+			if _, err := policy.ParseConfig(fin, fname, 4); err != nil {
+				t.Fatal(err)
+			}
+		})
+	}
+}
+
 // Regression test for CVE-2025-24369
 func TestCVE2025_24369(t *testing.T) {
 	pol := loadPolicies(t, "")
@@ -167,6 +186,7 @@ func TestCookieSettings(t *testing.T) {
 	}
 
 	if resp.StatusCode != http.StatusFound {
+		resp.Write(os.Stderr)
 		t.Errorf("wanted %d, got: %d", http.StatusFound, resp.StatusCode)
 	}
 
@@ -235,3 +255,141 @@ func TestCheckDefaultDifficultyMatchesPolicy(t *testing.T) {
 		})
 	}
 }
+
+func TestBasePrefix(t *testing.T) {
+	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		fmt.Fprintln(w, "OK")
+	})
+
+	testCases := []struct {
+		name       string
+		basePrefix string
+		path       string
+		expected   string
+	}{
+		{
+			name:       "no prefix",
+			basePrefix: "",
+			path:       "/.within.website/x/cmd/anubis/api/make-challenge",
+			expected:   "/.within.website/x/cmd/anubis/api/make-challenge",
+		},
+		{
+			name:       "with prefix",
+			basePrefix: "/myapp",
+			path:       "/myapp/.within.website/x/cmd/anubis/api/make-challenge",
+			expected:   "/myapp/.within.website/x/cmd/anubis/api/make-challenge",
+		},
+		{
+			name:       "with prefix and trailing slash",
+			basePrefix: "/myapp/",
+			path:       "/myapp/.within.website/x/cmd/anubis/api/make-challenge",
+			expected:   "/myapp/.within.website/x/cmd/anubis/api/make-challenge",
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			// Reset the global BasePrefix before each test
+			anubis.BasePrefix = ""
+
+			pol := loadPolicies(t, "")
+			pol.DefaultDifficulty = 4
+
+			srv := spawnAnubis(t, Options{
+				Next:       h,
+				Policy:     pol,
+				BasePrefix: tc.basePrefix,
+			})
+
+			ts := httptest.NewServer(internal.RemoteXRealIP(true, "tcp", srv))
+			defer ts.Close()
+
+			// Test API endpoint with prefix
+			resp, err := ts.Client().Post(ts.URL+tc.path, "", nil)
+			if err != nil {
+				t.Fatalf("can't request challenge: %v", err)
+			}
+			defer resp.Body.Close()
+
+			if resp.StatusCode != http.StatusOK {
+				t.Errorf("expected status code %d, got: %d", http.StatusOK, resp.StatusCode)
+			}
+
+			var chall challenge
+			if err := json.NewDecoder(resp.Body).Decode(&chall); err != nil {
+				t.Fatalf("can't read challenge response body: %v", err)
+			}
+
+			if chall.Challenge == "" {
+				t.Errorf("expected non-empty challenge")
+			}
+
+			// Test cookie path when passing challenge
+			// Find a nonce that produces a hash with the required number of leading zeros
+			nonce := 0
+			var calculated string
+			for {
+				calcString := fmt.Sprintf("%s%d", chall.Challenge, nonce)
+				calculated = internal.SHA256sum(calcString)
+				if strings.HasPrefix(calculated, strings.Repeat("0", pol.DefaultDifficulty)) {
+					break
+				}
+				nonce++
+			}
+			elapsedTime := 420
+			redir := "/"
+
+			cli := ts.Client()
+			cli.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+				return http.ErrUseLastResponse
+			}
+
+			// Construct the correct path for pass-challenge
+			passChallengePath := tc.path
+			passChallengePath = passChallengePath[:strings.LastIndex(passChallengePath, "/")+1] + "pass-challenge"
+
+			req, err := http.NewRequest(http.MethodGet, ts.URL+passChallengePath, nil)
+			if err != nil {
+				t.Fatalf("can't make request: %v", err)
+			}
+
+			q := req.URL.Query()
+			q.Set("response", calculated)
+			q.Set("nonce", fmt.Sprint(nonce))
+			q.Set("redir", redir)
+			q.Set("elapsedTime", fmt.Sprint(elapsedTime))
+			req.URL.RawQuery = q.Encode()
+
+			resp, err = cli.Do(req)
+			if err != nil {
+				t.Fatalf("can't do challenge passing: %v", err)
+			}
+
+			if resp.StatusCode != http.StatusFound {
+				t.Errorf("wanted %d, got: %d", http.StatusFound, resp.StatusCode)
+			}
+
+			// Check cookie path
+			var ckie *http.Cookie
+			for _, cookie := range resp.Cookies() {
+				if cookie.Name == anubis.CookieName {
+					ckie = cookie
+					break
+				}
+			}
+			if ckie == nil {
+				t.Errorf("Cookie %q not found", anubis.CookieName)
+				return
+			}
+
+			expectedPath := "/"
+			if tc.basePrefix != "" {
+				expectedPath = strings.TrimSuffix(tc.basePrefix, "/") + "/"
+			}
+
+			if ckie.Path != expectedPath {
+				t.Errorf("cookie path is wrong, wanted %s, got: %s", expectedPath, ckie.Path)
+			}
+		})
+	}
+}

lib/policy/bot.go

@@ -2,31 +2,18 @@ package policy
 
 import (
 	"fmt"
-	"regexp"
 
 	"github.com/TecharoHQ/anubis/internal"
 	"github.com/TecharoHQ/anubis/lib/policy/config"
-	"github.com/yl2chen/cidranger"
 )
 
 type Bot struct {
 	Name      string
-	UserAgent *regexp.Regexp
-	Path      *regexp.Regexp
-	Action    config.Rule `json:"action"`
+	Action    config.Rule
 	Challenge *config.ChallengeRules
-	Ranger    cidranger.Ranger
+	Rules     Checker
 }
 
-func (b Bot) Hash() (string, error) {
-	var pathRex string
-	if b.Path != nil {
-		pathRex = b.Path.String()
-	}
-	var userAgentRex string
-	if b.UserAgent != nil {
-		userAgentRex = b.UserAgent.String()
-	}
-
-	return internal.SHA256sum(fmt.Sprintf("%s::%s::%s", b.Name, pathRex, userAgentRex)), nil
+func (b Bot) Hash() string {
+	return internal.SHA256sum(fmt.Sprintf("%s::%s", b.Name, b.Rules.Hash()))
 }

lib/policy/checker.go

@@ -0,0 +1,201 @@
+package policy
+
+import (
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"regexp"
+	"strings"
+
+	"github.com/TecharoHQ/anubis/internal"
+	"github.com/yl2chen/cidranger"
+)
+
+var (
+	ErrMisconfiguration = errors.New("[unexpected] policy: administrator misconfiguration")
+)
+
+type Checker interface {
+	Check(*http.Request) (bool, error)
+	Hash() string
+}
+
+type CheckerList []Checker
+
+func (cl CheckerList) Check(r *http.Request) (bool, error) {
+	for _, c := range cl {
+		ok, err := c.Check(r)
+		if err != nil {
+			return ok, err
+		}
+		if ok {
+			return ok, nil
+		}
+	}
+
+	return false, nil
+}
+
+func (cl CheckerList) Hash() string {
+	var sb strings.Builder
+
+	for _, c := range cl {
+		fmt.Fprintln(&sb, c.Hash())
+	}
+
+	return internal.SHA256sum(sb.String())
+}
+
+type RemoteAddrChecker struct {
+	ranger cidranger.Ranger
+	hash   string
+}
+
+func NewRemoteAddrChecker(cidrs []string) (Checker, error) {
+	ranger := cidranger.NewPCTrieRanger()
+	var sb strings.Builder
+
+	for _, cidr := range cidrs {
+		_, rng, err := net.ParseCIDR(cidr)
+		if err != nil {
+			return nil, fmt.Errorf("%w: range %s not parsing: %w", ErrMisconfiguration, cidr, err)
+		}
+
+		ranger.Insert(cidranger.NewBasicRangerEntry(*rng))
+		fmt.Fprintln(&sb, cidr)
+	}
+
+	return &RemoteAddrChecker{
+		ranger: ranger,
+		hash:   internal.SHA256sum(sb.String()),
+	}, nil
+}
+
+func (rac *RemoteAddrChecker) Check(r *http.Request) (bool, error) {
+	host := r.Header.Get("X-Real-Ip")
+	if host == "" {
+		return false, fmt.Errorf("%w: header X-Real-Ip is not set", ErrMisconfiguration)
+	}
+
+	addr := net.ParseIP(host)
+	if addr == nil {
+		return false, fmt.Errorf("%w: %s is not an IP address", ErrMisconfiguration, host)
+	}
+
+	ok, err := rac.ranger.Contains(addr)
+	if err != nil {
+		return false, err
+	}
+
+	if ok {
+		return true, nil
+	}
+
+	return false, nil
+}
+
+func (rac *RemoteAddrChecker) Hash() string {
+	return rac.hash
+}
+
+type HeaderMatchesChecker struct {
+	header string
+	regexp *regexp.Regexp
+	hash   string
+}
+
+func NewUserAgentChecker(rexStr string) (Checker, error) {
+	return NewHeaderMatchesChecker("User-Agent", rexStr)
+}
+
+func NewHeaderMatchesChecker(header, rexStr string) (Checker, error) {
+	rex, err := regexp.Compile(strings.TrimSpace(rexStr))
+	if err != nil {
+		return nil, fmt.Errorf("%w: regex %s failed parse: %w", ErrMisconfiguration, rexStr, err)
+	}
+	return &HeaderMatchesChecker{strings.TrimSpace(header), rex, internal.SHA256sum(header + ": " + rexStr)}, nil
+}
+
+func (hmc *HeaderMatchesChecker) Check(r *http.Request) (bool, error) {
+	if hmc.regexp.MatchString(r.Header.Get(hmc.header)) {
+		return true, nil
+	}
+
+	return false, nil
+}
+
+func (hmc *HeaderMatchesChecker) Hash() string {
+	return hmc.hash
+}
+
+type PathChecker struct {
+	regexp *regexp.Regexp
+	hash   string
+}
+
+func NewPathChecker(rexStr string) (Checker, error) {
+	rex, err := regexp.Compile(strings.TrimSpace(rexStr))
+	if err != nil {
+		return nil, fmt.Errorf("%w: regex %s failed parse: %w", ErrMisconfiguration, rexStr, err)
+	}
+	return &PathChecker{rex, internal.SHA256sum(rexStr)}, nil
+}
+
+func (pc *PathChecker) Check(r *http.Request) (bool, error) {
+	if pc.regexp.MatchString(r.URL.Path) {
+		return true, nil
+	}
+
+	return false, nil
+}
+
+func (pc *PathChecker) Hash() string {
+	return pc.hash
+}
+
+func NewHeaderExistsChecker(key string) Checker {
+	return headerExistsChecker{strings.TrimSpace(key)}
+}
+
+type headerExistsChecker struct {
+	header string
+}
+
+func (hec headerExistsChecker) Check(r *http.Request) (bool, error) {
+	if r.Header.Get(hec.header) != "" {
+		return true, nil
+	}
+
+	return false, nil
+}
+
+func (hec headerExistsChecker) Hash() string {
+	return internal.SHA256sum(hec.header)
+}
+
+func NewHeadersChecker(headermap map[string]string) (Checker, error) {
+	var result CheckerList
+	var errs []error
+
+	for key, rexStr := range headermap {
+		if rexStr == ".*" {
+			result = append(result, headerExistsChecker{strings.TrimSpace(key)})
+			continue
+		}
+
+		rex, err := regexp.Compile(strings.TrimSpace(rexStr))
+		if err != nil {
+			errs = append(errs, fmt.Errorf("while compiling header %s regex %s: %w", key, rexStr, err))
+			continue
+		}
+
+		result = append(result, &HeaderMatchesChecker{key, rex, internal.SHA256sum(key + ": " + rexStr)})
+	}
+
+	if len(errs) != 0 {
+		return nil, errors.Join(errs...)
+	}
+
+	return result, nil
+}

lib/policy/checker_test.go

@@ -0,0 +1,200 @@
+package policy
+
+import (
+	"errors"
+	"net/http"
+	"testing"
+)
+
+func TestRemoteAddrChecker(t *testing.T) {
+	for _, tt := range []struct {
+		name  string
+		cidrs []string
+		ip    string
+		ok    bool
+		err   error
+	}{
+		{
+			name:  "match_ipv4",
+			cidrs: []string{"0.0.0.0/0"},
+			ip:    "1.1.1.1",
+			ok:    true,
+			err:   nil,
+		},
+		{
+			name:  "match_ipv6",
+			cidrs: []string{"::/0"},
+			ip:    "cafe:babe::",
+			ok:    true,
+			err:   nil,
+		},
+		{
+			name:  "not_match_ipv4",
+			cidrs: []string{"1.1.1.1/32"},
+			ip:    "1.1.1.2",
+			ok:    false,
+			err:   nil,
+		},
+		{
+			name:  "not_match_ipv6",
+			cidrs: []string{"cafe:babe::/128"},
+			ip:    "cafe:babe:4::/128",
+			ok:    false,
+			err:   nil,
+		},
+		{
+			name:  "no_ip_set",
+			cidrs: []string{"::/0"},
+			ok:    false,
+			err:   ErrMisconfiguration,
+		},
+		{
+			name:  "invalid_ip",
+			cidrs: []string{"::/0"},
+			ip:    "According to all natural laws of aviation",
+			ok:    false,
+			err:   ErrMisconfiguration,
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			rac, err := NewRemoteAddrChecker(tt.cidrs)
+			if err != nil && !errors.Is(err, tt.err) {
+				t.Fatalf("creating RemoteAddrChecker failed: %v", err)
+			}
+
+			r, err := http.NewRequest(http.MethodGet, "/", nil)
+			if err != nil {
+				t.Fatalf("can't make request: %v", err)
+			}
+
+			if tt.ip != "" {
+				r.Header.Add("X-Real-Ip", tt.ip)
+			}
+
+			ok, err := rac.Check(r)
+
+			if tt.ok != ok {
+				t.Errorf("ok: %v, wanted: %v", ok, tt.ok)
+			}
+
+			if err != nil && tt.err != nil && !errors.Is(err, tt.err) {
+				t.Errorf("err: %v, wanted: %v", err, tt.err)
+			}
+		})
+	}
+}
+
+func TestHeaderMatchesChecker(t *testing.T) {
+	for _, tt := range []struct {
+		name           string
+		header         string
+		rexStr         string
+		reqHeaderKey   string
+		reqHeaderValue string
+		ok             bool
+		err            error
+	}{
+		{
+			name:           "match",
+			header:         "Cf-Worker",
+			rexStr:         ".*",
+			reqHeaderKey:   "Cf-Worker",
+			reqHeaderValue: "true",
+			ok:             true,
+			err:            nil,
+		},
+		{
+			name:           "not_match",
+			header:         "Cf-Worker",
+			rexStr:         "false",
+			reqHeaderKey:   "Cf-Worker",
+			reqHeaderValue: "true",
+			ok:             false,
+			err:            nil,
+		},
+		{
+			name:           "not_present",
+			header:         "Cf-Worker",
+			rexStr:         "foobar",
+			reqHeaderKey:   "Something-Else",
+			reqHeaderValue: "true",
+			ok:             false,
+			err:            nil,
+		},
+		{
+			name:   "invalid_regex",
+			rexStr: "a(b",
+			err:    ErrMisconfiguration,
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			hmc, err := NewHeaderMatchesChecker(tt.header, tt.rexStr)
+			if err != nil && !errors.Is(err, tt.err) {
+				t.Fatalf("creating HeaderMatchesChecker failed")
+			}
+
+			if tt.err != nil && hmc == nil {
+				return
+			}
+
+			r, err := http.NewRequest(http.MethodGet, "/", nil)
+			if err != nil {
+				t.Fatalf("can't make request: %v", err)
+			}
+
+			r.Header.Set(tt.reqHeaderKey, tt.reqHeaderValue)
+
+			ok, err := hmc.Check(r)
+
+			if tt.ok != ok {
+				t.Errorf("ok: %v, wanted: %v", ok, tt.ok)
+			}
+
+			if err != nil && tt.err != nil && !errors.Is(err, tt.err) {
+				t.Errorf("err: %v, wanted: %v", err, tt.err)
+			}
+		})
+	}
+}
+
+func TestHeaderExistsChecker(t *testing.T) {
+	for _, tt := range []struct {
+		name      string
+		header    string
+		reqHeader string
+		ok        bool
+	}{
+		{
+			name:      "match",
+			header:    "Authorization",
+			reqHeader: "Authorization",
+			ok:        true,
+		},
+		{
+			name:      "not_match",
+			header:    "Authorization",
+			reqHeader: "Authentication",
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			hec := headerExistsChecker{tt.header}
+
+			r, err := http.NewRequest(http.MethodGet, "/", nil)
+			if err != nil {
+				t.Fatalf("can't make request: %v", err)
+			}
+
+			r.Header.Set(tt.reqHeader, "hunter2")
+
+			ok, err := hec.Check(r)
+
+			if tt.ok != ok {
+				t.Errorf("ok: %v, wanted: %v", ok, tt.ok)
+			}
+
+			if err != nil {
+				t.Errorf("err: %v", err)
+			}
+		})
+	}
+}

lib/policy/checkresult.go

@@ -1,4 +1,4 @@
-package lib
+package policy
 
 import (
 	"log/slog"
@@ -16,10 +16,3 @@ func (cr CheckResult) LogValue() slog.Value {
 		slog.String("name", cr.Name),
 		slog.String("rule", string(cr.Rule)))
 }
-
-func cr(name string, rule config.Rule) CheckResult {
-	return CheckResult{
-		Name: name,
-		Rule: rule,
-	}
-}

lib/policy/config/config.go

@@ -3,19 +3,31 @@ package config
 import (
 	"errors"
 	"fmt"
+	"io"
+	"io/fs"
 	"net"
+	"os"
 	"regexp"
+	"strings"
+
+	"github.com/TecharoHQ/anubis/data"
+	"k8s.io/apimachinery/pkg/util/yaml"
 )
 
 var (
 	ErrNoBotRulesDefined                 = errors.New("config: must define at least one (1) bot rule")
 	ErrBotMustHaveName                   = errors.New("config.Bot: must set name")
-	ErrBotMustHaveUserAgentOrPath        = errors.New("config.Bot: must set either user_agent_regex, path_regex, or remote_addresses")
+	ErrBotMustHaveUserAgentOrPath        = errors.New("config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses")
 	ErrBotMustHaveUserAgentOrPathNotBoth = errors.New("config.Bot: must set either user_agent_regex, path_regex, and not both")
 	ErrUnknownAction                     = errors.New("config.Bot: unknown action")
 	ErrInvalidUserAgentRegex             = errors.New("config.Bot: invalid user agent regex")
 	ErrInvalidPathRegex                  = errors.New("config.Bot: invalid path regex")
+	ErrInvalidHeadersRegex               = errors.New("config.Bot: invalid headers regex")
 	ErrInvalidCIDR                       = errors.New("config.Bot: invalid CIDR")
+	ErrRegexEndsWithNewline              = errors.New("config.Bot: regular expression ends with newline (try >- instead of > in yaml)")
+	ErrInvalidImportStatement            = errors.New("config.ImportStatement: invalid source file")
+	ErrCantSetBotAndImportValuesAtOnce   = errors.New("config.BotOrImport: can't set bot rules and import values at the same time")
+	ErrMustSetBotOrImportRules           = errors.New("config.BotOrImport: rule definition is invalid, you must set either bot rules or an import statement, not both")
 )
 
 type Rule string
@@ -37,12 +49,31 @@ const (
 )
 
 type BotConfig struct {
-	Name           string          `json:"name"`
-	UserAgentRegex *string         `json:"user_agent_regex"`
-	PathRegex      *string         `json:"path_regex"`
-	Action         Rule            `json:"action"`
-	RemoteAddr     []string        `json:"remote_addresses"`
-	Challenge      *ChallengeRules `json:"challenge,omitempty"`
+	Name           string            `json:"name"`
+	UserAgentRegex *string           `json:"user_agent_regex"`
+	PathRegex      *string           `json:"path_regex"`
+	HeadersRegex   map[string]string `json:"headers_regex"`
+	Action         Rule              `json:"action"`
+	RemoteAddr     []string          `json:"remote_addresses"`
+	Challenge      *ChallengeRules   `json:"challenge,omitempty"`
+}
+
+func (b BotConfig) Zero() bool {
+	for _, cond := range []bool{
+		b.Name != "",
+		b.UserAgentRegex != nil,
+		b.PathRegex != nil,
+		len(b.HeadersRegex) != 0,
+		b.Action != "",
+		len(b.RemoteAddr) != 0,
+		b.Challenge != nil,
+	} {
+		if cond {
+			return false
+		}
+	}
+
+	return true
 }
 
 func (b BotConfig) Valid() error {
@@ -52,7 +83,7 @@ func (b BotConfig) Valid() error {
 		errs = append(errs, ErrBotMustHaveName)
 	}
 
-	if b.UserAgentRegex == nil && b.PathRegex == nil && len(b.RemoteAddr) == 0 {
+	if b.UserAgentRegex == nil && b.PathRegex == nil && len(b.RemoteAddr) == 0 && len(b.HeadersRegex) == 0 {
 		errs = append(errs, ErrBotMustHaveUserAgentOrPath)
 	}
 
@@ -61,17 +92,41 @@ func (b BotConfig) Valid() error {
 	}
 
 	if b.UserAgentRegex != nil {
+		if strings.HasSuffix(*b.UserAgentRegex, "\n") {
+			errs = append(errs, fmt.Errorf("%w: user agent regex: %q", ErrRegexEndsWithNewline, *b.UserAgentRegex))
+		}
+
 		if _, err := regexp.Compile(*b.UserAgentRegex); err != nil {
 			errs = append(errs, ErrInvalidUserAgentRegex, err)
 		}
 	}
 
 	if b.PathRegex != nil {
+		if strings.HasSuffix(*b.PathRegex, "\n") {
+			errs = append(errs, fmt.Errorf("%w: path regex: %q", ErrRegexEndsWithNewline, *b.PathRegex))
+		}
+
 		if _, err := regexp.Compile(*b.PathRegex); err != nil {
 			errs = append(errs, ErrInvalidPathRegex, err)
 		}
 	}
 
+	if len(b.HeadersRegex) > 0 {
+		for name, expr := range b.HeadersRegex {
+			if name == "" {
+				continue
+			}
+
+			if strings.HasSuffix(expr, "\n") {
+				errs = append(errs, fmt.Errorf("%w: header %s regex: %q", ErrRegexEndsWithNewline, name, expr))
+			}
+
+			if _, err := regexp.Compile(expr); err != nil {
+				errs = append(errs, ErrInvalidHeadersRegex, err)
+			}
+		}
+	}
+
 	if len(b.RemoteAddr) > 0 {
 		for _, cidr := range b.RemoteAddr {
 			if _, _, err := net.ParseCIDR(cidr); err != nil {
@@ -137,9 +192,147 @@ func (cr ChallengeRules) Valid() error {
 	return nil
 }
 
+type ImportStatement struct {
+	Import string `json:"import"`
+	Bots   []BotConfig
+}
+
+func (is *ImportStatement) open() (fs.File, error) {
+	if strings.HasPrefix(is.Import, "(data)/") {
+		fname := strings.TrimPrefix(is.Import, "(data)/")
+		fin, err := data.BotPolicies.Open(fname)
+		return fin, err
+	}
+
+	return os.Open(is.Import)
+}
+
+func (is *ImportStatement) load() error {
+	fin, err := is.open()
+	if err != nil {
+		return fmt.Errorf("can't open %s: %w", is.Import, err)
+	}
+	defer fin.Close()
+
+	var result []BotConfig
+
+	if err := yaml.NewYAMLToJSONDecoder(fin).Decode(&result); err != nil {
+		return fmt.Errorf("can't parse %s: %w", is.Import, err)
+	}
+
+	var errs []error
+
+	for _, b := range result {
+		if err := b.Valid(); err != nil {
+			errs = append(errs, err)
+		}
+	}
+
+	if len(errs) != 0 {
+		return fmt.Errorf("config %s is not valid:\n%w", is.Import, errors.Join(errs...))
+	}
+
+	is.Bots = result
+
+	return nil
+}
+
+func (is *ImportStatement) Valid() error {
+	return is.load()
+}
+
+type BotOrImport struct {
+	*BotConfig       `json:",inline"`
+	*ImportStatement `json:",inline"`
+}
+
+func (boi *BotOrImport) Valid() error {
+	if boi.BotConfig != nil && boi.ImportStatement != nil {
+		return ErrCantSetBotAndImportValuesAtOnce
+	}
+
+	if boi.BotConfig != nil {
+		return boi.BotConfig.Valid()
+	}
+
+	if boi.ImportStatement != nil {
+		return boi.ImportStatement.Valid()
+	}
+
+	return ErrMustSetBotOrImportRules
+}
+
+type fileConfig struct {
+	Bots  []BotOrImport `json:"bots"`
+	DNSBL bool          `json:"dnsbl"`
+}
+
+func (c fileConfig) Valid() error {
+	var errs []error
+
+	if len(c.Bots) == 0 {
+		errs = append(errs, ErrNoBotRulesDefined)
+	}
+
+	for _, b := range c.Bots {
+		if err := b.Valid(); err != nil {
+			errs = append(errs, err)
+		}
+	}
+
+	if len(errs) != 0 {
+		return fmt.Errorf("config is not valid:\n%w", errors.Join(errs...))
+	}
+
+	return nil
+}
+
+func Load(fin io.Reader, fname string) (*Config, error) {
+	var c fileConfig
+	if err := yaml.NewYAMLToJSONDecoder(fin).Decode(&c); err != nil {
+		return nil, fmt.Errorf("can't parse policy config YAML %s: %w", fname, err)
+	}
+
+	if err := c.Valid(); err != nil {
+		return nil, err
+	}
+
+	result := &Config{
+		DNSBL: c.DNSBL,
+	}
+
+	var validationErrs []error
+
+	for _, boi := range c.Bots {
+		if boi.ImportStatement != nil {
+			if err := boi.load(); err != nil {
+				validationErrs = append(validationErrs, err)
+				continue
+			}
+
+			result.Bots = append(result.Bots, boi.ImportStatement.Bots...)
+		}
+
+		if boi.BotConfig != nil {
+			if err := boi.BotConfig.Valid(); err != nil {
+				validationErrs = append(validationErrs, err)
+				continue
+			}
+
+			result.Bots = append(result.Bots, *boi.BotConfig)
+		}
+	}
+
+	if len(validationErrs) > 0 {
+		return nil, fmt.Errorf("errors validating policy config %s: %w", fname, errors.Join(validationErrs...))
+	}
+
+	return result, nil
+}
+
 type Config struct {
-	Bots  []BotConfig `json:"bots"`
-	DNSBL bool        `json:"dnsbl"`
+	Bots  []BotConfig
+	DNSBL bool
 }
 
 func (c Config) Valid() error {

lib/policy/config/config_test.go

@@ -1,11 +1,14 @@
 package config
 
 import (
-	"encoding/json"
 	"errors"
+	"io/fs"
 	"os"
 	"path/filepath"
 	"testing"
+
+	"github.com/TecharoHQ/anubis/data"
+	"k8s.io/apimachinery/pkg/util/yaml"
 )
 
 func p[V any](v V) *V { return &v }
@@ -87,6 +90,18 @@ func TestBotValid(t *testing.T) {
 			},
 			err: ErrInvalidPathRegex,
 		},
+		{
+			name: "invalid headers regex",
+			bot: BotConfig{
+				Name:   "mozilla-ua",
+				Action: RuleChallenge,
+				HeadersRegex: map[string]string{
+					"Content-Type": "a(b",
+				},
+				PathRegex: p("a(b"),
+			},
+			err: ErrInvalidHeadersRegex,
+		},
 		{
 			name: "challenge difficulty too low",
 			bot: BotConfig{
@@ -206,13 +221,69 @@ func TestConfigValidKnownGood(t *testing.T) {
 			}
 			defer fin.Close()
 
-			var c Config
-			if err := json.NewDecoder(fin).Decode(&c); err != nil {
-				t.Fatalf("can't decode file: %v", err)
+			c, err := Load(fin, st.Name())
+			if err != nil {
+				t.Fatal(err)
 			}
 
 			if err := c.Valid(); err != nil {
-				t.Fatal(err)
+				t.Error(err)
+			}
+
+			if len(c.Bots) == 0 {
+				t.Error("wanted more than 0 bots, got zero")
+			}
+		})
+	}
+}
+
+func TestImportStatement(t *testing.T) {
+	type testCase struct {
+		name       string
+		importPath string
+		err        error
+	}
+
+	var tests []testCase
+
+	for _, folderName := range []string{
+		"apps",
+		"bots",
+		"common",
+		"crawlers",
+	} {
+		if err := fs.WalkDir(data.BotPolicies, folderName, func(path string, d fs.DirEntry, err error) error {
+			if err != nil {
+				return err
+			}
+			if d.IsDir() {
+				return nil
+			}
+
+			tests = append(tests, testCase{
+				name:       "(data)/" + path,
+				importPath: "(data)/" + path,
+				err:        nil,
+			})
+
+			return nil
+		}); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			is := &ImportStatement{
+				Import: tt.importPath,
+			}
+
+			if err := is.Valid(); err != nil {
+				t.Errorf("validation error: %v", err)
+			}
+
+			if len(is.Bots) == 0 {
+				t.Error("wanted bot definitions, but got none")
 			}
 		})
 	}
@@ -233,8 +304,8 @@ func TestConfigValidBad(t *testing.T) {
 			}
 			defer fin.Close()
 
-			var c Config
-			if err := json.NewDecoder(fin).Decode(&c); err != nil {
+			var c fileConfig
+			if err := yaml.NewYAMLToJSONDecoder(fin).Decode(&c); err != nil {
 				t.Fatalf("can't decode file: %v", err)
 			}
 
@@ -246,3 +317,49 @@ func TestConfigValidBad(t *testing.T) {
 		})
 	}
 }
+
+func TestBotConfigZero(t *testing.T) {
+	var b BotConfig
+	if !b.Zero() {
+		t.Error("zero value BotConfig is not zero value")
+	}
+
+	b.Name = "hi"
+	if b.Zero() {
+		t.Error("BotConfig with name is zero value")
+	}
+
+	b.UserAgentRegex = p(".*")
+	if b.Zero() {
+		t.Error("BotConfig with user agent regex is zero value")
+	}
+
+	b.PathRegex = p(".*")
+	if b.Zero() {
+		t.Error("BotConfig with path regex is zero value")
+	}
+
+	b.HeadersRegex = map[string]string{"hi": "there"}
+	if b.Zero() {
+		t.Error("BotConfig with headers regex is zero value")
+	}
+
+	b.Action = RuleAllow
+	if b.Zero() {
+		t.Error("BotConfig with action is zero value")
+	}
+
+	b.RemoteAddr = []string{"::/0"}
+	if b.Zero() {
+		t.Error("BotConfig with remote addresses is zero value")
+	}
+
+	b.Challenge = &ChallengeRules{
+		Difficulty: 4,
+		ReportAs:   4,
+		Algorithm:  AlgorithmFast,
+	}
+	if b.Zero() {
+		t.Error("BotConfig with challenge rules is zero value")
+	}
+}

lib/policy/config/testdata/bad/badregexes.json

@@ -9,6 +9,13 @@
       "name": "user-agent-bad",
       "user_agent_regex": "a(b",
       "action": "DENY"
+    },
+    {
+      "name": "headers-bad",
+      "headers": {
+        "Accept-Encoding": "a(b"
+      },
+      "action": "DENY"
     }
   ]
 }

lib/policy/config/testdata/bad/badregexes.yaml

@@ -0,0 +1,7 @@
+bots:
+- name: path-bad
+  path_regex: "a(b"
+  action: DENY
+- name: user-agent-bad
+  user_agent_regex: "a(b"
+  action: DENY

lib/policy/config/testdata/bad/import_and_bot.json

@@ -0,0 +1,10 @@
+{
+  "bots": [
+    {
+      "import": "(data)/bots/ai-robots-txt.yaml",
+      "name": "generic-browser",
+      "user_agent_regex": "Mozilla|Opera\n",
+      "action": "CHALLENGE"
+    }
+  ]
+}

lib/policy/config/testdata/bad/import_and_bot.yaml

@@ -0,0 +1,6 @@
+bots:
+- import: (data)/bots/ai-robots-txt.yaml
+  name: generic-browser
+  user_agent_regex: >
+    Mozilla|Opera
+  action: CHALLENGE

lib/policy/config/testdata/bad/import_invalid_file.json

@@ -0,0 +1,7 @@
+{
+  "bots": [
+    {
+      "import": "(data)/does-not-exist-fake-file.yaml"
+    }
+  ]
+}

lib/policy/config/testdata/bad/import_invalid_file.yaml

@@ -0,0 +1,2 @@
+bots:
+- import: (data)/does-not-exist-fake-file.yaml

lib/policy/config/testdata/bad/invalid.yaml

@@ -0,0 +1 @@
+bots: []

lib/policy/config/testdata/bad/nobots.yaml

@@ -0,0 +1 @@
+{}

lib/policy/config/testdata/bad/regex_ends_newline.json

@@ -0,0 +1,21 @@
+{
+  "bots": [
+    {
+      "name": "user-agent-ends-newline",
+      "user_agent_regex": "Mozilla\n",
+      "action": "CHALLENGE"
+    },
+    {
+      "name": "path-ends-newline",
+      "path_regex": "^/evil/.*$\n",
+      "action": "CHALLENGE"
+    },
+    {
+      "name": "headers-ends-newline",
+      "headers_regex": {
+        "CF-Worker": ".*\n"
+      },
+      "action": "CHALLENGE"
+    }
+  ]
+}

lib/policy/config/testdata/bad/regex_ends_newline.yaml

@@ -0,0 +1,17 @@
+bots:
+- name: user-agent-ends-newline
+  # Subtle bug: this ends with a newline
+  user_agent_regex: >
+    Mozilla
+  action: CHALLENGE
+- name: path-ends-newline
+  # Subtle bug: this ends with a newline
+  path_regex: >
+    ^/evil/.*$
+  action: CHALLENGE
+- name: headers-ends-newline
+  # Subtle bug: this ends with a newline
+  headers_regex:
+    CF-Worker: >
+      .*
+  action: CHALLENGE

lib/policy/config/testdata/good/allow_everyone.yaml

@@ -0,0 +1,6 @@
+bots:
+- name: everyones-invited
+  remote_addresses:
+    - "0.0.0.0/0"
+    - "::/0"
+  action: ALLOW

lib/policy/config/testdata/good/block_cf_workers.json

@@ -0,0 +1,12 @@
+{
+  "bots": [
+    {
+      "name": "Cloudflare Workers",
+      "headers_regex": {
+        "CF-Worker": ".*"
+      },
+      "action": "DENY"
+    }
+  ],
+  "dnsbl": false
+}

lib/policy/config/testdata/good/block_cf_workers.yaml

@@ -0,0 +1,5 @@
+bots:
+  - name: cloudflare-workers
+    headers_regex:
+      CF-Worker: .*
+    action: DENY

lib/policy/config/testdata/good/challengemozilla.yaml

@@ -0,0 +1,4 @@
+bots:
+- name: generic-browser
+  user_agent_regex: Mozilla
+  action: CHALLENGE

lib/policy/config/testdata/good/everything_blocked.yaml

@@ -0,0 +1,4 @@
+bots:
+- name: everything
+  user_agent_regex: .*
+  action: DENY

lib/policy/config/testdata/good/import_filesystem.json

@@ -0,0 +1,7 @@
+{
+  "bots": [
+    {
+      "import": "./testdata/hack-test.json"
+    }
+  ]
+}

lib/policy/config/testdata/good/import_filesystem.yaml

@@ -0,0 +1,2 @@
+bots:
+- import: ./testdata/hack-test.yaml

lib/policy/config/testdata/good/import_keep_internet_working.json

@@ -0,0 +1,7 @@
+{
+  "bots": [
+    {
+      "import": "(data)/common/keep-internet-working.yaml"
+    }
+  ]
+}

lib/policy/config/testdata/good/import_keep_internet_working.yaml

@@ -0,0 +1,2 @@
+bots:
+- import: (data)/common/keep-internet-working.yaml

lib/policy/config/testdata/hack-test.json

@@ -0,0 +1,9 @@
+[
+  {
+    "name": "ipv6-ula",
+    "action": "ALLOW",
+    "remote_addresses": [
+      "fc00::/7"
+    ]
+  }
+]

lib/policy/config/testdata/hack-test.yaml

@@ -0,0 +1,3 @@
+- name: well-known
+  path_regex: ^/.well-known/.*$
+  action: ALLOW

lib/policy/policy.go

@@ -1,48 +1,40 @@
 package policy
 
 import (
-	"encoding/json"
 	"errors"
 	"fmt"
 	"io"
-	"net"
-	"regexp"
 
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/prometheus/client_golang/prometheus/promauto"
-	"github.com/yl2chen/cidranger"
 
 	"github.com/TecharoHQ/anubis/lib/policy/config"
 )
 
 var (
-	PolicyApplications = promauto.NewCounterVec(prometheus.CounterOpts{
+	Applications = promauto.NewCounterVec(prometheus.CounterOpts{
 		Name: "anubis_policy_results",
 		Help: "The results of each policy rule",
 	}, []string{"rule", "action"})
 )
 
 type ParsedConfig struct {
-	orig config.Config
+	orig *config.Config
 
 	Bots              []Bot
 	DNSBL             bool
 	DefaultDifficulty int
 }
 
-func NewParsedConfig(orig config.Config) *ParsedConfig {
+func NewParsedConfig(orig *config.Config) *ParsedConfig {
 	return &ParsedConfig{
 		orig: orig,
 	}
 }
 
 func ParseConfig(fin io.Reader, fname string, defaultDifficulty int) (*ParsedConfig, error) {
-	var c config.Config
-	if err := json.NewDecoder(fin).Decode(&c); err != nil {
-		return nil, fmt.Errorf("can't parse policy config JSON %s: %w", fname, err)
-	}
-
-	if err := c.Valid(); err != nil {
+	c, err := config.Load(fin, fname)
+	if err != nil {
 		return nil, err
 	}
 
@@ -62,36 +54,41 @@ func ParseConfig(fin io.Reader, fname string, defaultDifficulty int) (*ParsedCon
 			Action: b.Action,
 		}
 
+		cl := CheckerList{}
+
 		if len(b.RemoteAddr) > 0 {
-			parsedBot.Ranger = cidranger.NewPCTrieRanger()
-
-			for _, cidr := range b.RemoteAddr {
-				_, rng, err := net.ParseCIDR(cidr)
-				if err != nil {
-					return nil, fmt.Errorf("[unexpected] range %s not parsing: %w", cidr, err)
-				}
-
-				parsedBot.Ranger.Insert(cidranger.NewBasicRangerEntry(*rng))
+			c, err := NewRemoteAddrChecker(b.RemoteAddr)
+			if err != nil {
+				validationErrs = append(validationErrs, fmt.Errorf("while processing rule %s remote addr set: %w", b.Name, err))
+			} else {
+				cl = append(cl, c)
 			}
 		}
 
 		if b.UserAgentRegex != nil {
-			userAgent, err := regexp.Compile(*b.UserAgentRegex)
+			c, err := NewUserAgentChecker(*b.UserAgentRegex)
 			if err != nil {
-				validationErrs = append(validationErrs, fmt.Errorf("while compiling user agent regexp: %w", err))
-				continue
+				validationErrs = append(validationErrs, fmt.Errorf("while processing rule %s user agent regex: %w", b.Name, err))
 			} else {
-				parsedBot.UserAgent = userAgent
+				cl = append(cl, c)
 			}
 		}
 
 		if b.PathRegex != nil {
-			path, err := regexp.Compile(*b.PathRegex)
+			c, err := NewPathChecker(*b.PathRegex)
 			if err != nil {
-				validationErrs = append(validationErrs, fmt.Errorf("while compiling path regexp: %w", err))
-				continue
+				validationErrs = append(validationErrs, fmt.Errorf("while processing rule %s path regex: %w", b.Name, err))
 			} else {
-				parsedBot.Path = path
+				cl = append(cl, c)
+			}
+		}
+
+		if len(b.HeadersRegex) > 0 {
+			c, err := NewHeadersChecker(b.HeadersRegex)
+			if err != nil {
+				validationErrs = append(validationErrs, fmt.Errorf("while processing rule %s headers regex map: %w", b.Name, err))
+			} else {
+				cl = append(cl, c)
 			}
 		}
 
@@ -108,6 +105,8 @@ func ParseConfig(fin io.Reader, fname string, defaultDifficulty int) (*ParsedCon
 			}
 		}
 
+		parsedBot.Rules = cl
+
 		result.Bots = append(result.Bots, parsedBot)
 	}
 

lib/policy/testdata/hack-test.json

@@ -0,0 +1,9 @@
+[
+  {
+    "name": "ipv6-ula",
+    "action": "ALLOW",
+    "remote_addresses": [
+      "fc00::/7"
+    ]
+  }
+]

lib/policy/testdata/hack-test.yaml

@@ -0,0 +1,3 @@
+- name: well-known
+  path_regex: ^/.well-known/.*$
+  action: ALLOW

lib/random.go

@@ -1,9 +0,0 @@
-package lib
-
-import (
-	"math/rand"
-)
-
-func randomJitter() bool {
-	return rand.Intn(100) > 10
-}

package.json

@@ -23,4 +23,4 @@
     "postcss-import-url": "^7.2.0",
     "postcss-url": "^10.1.3"
   }
-}
+}

test/.gitignore

@@ -0,0 +1,2 @@
+*.sock
+*.pem

test/cmd/relayd/main.go

@@ -0,0 +1,124 @@
+package main
+
+import (
+	"context"
+	"flag"
+	"log"
+	"log/slog"
+	"net"
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"github.com/TecharoHQ/anubis/internal"
+	"github.com/facebookgo/flagenv"
+	"github.com/google/uuid"
+)
+
+var (
+	bind      = flag.String("bind", ":3004", "port to listen on")
+	certDir   = flag.String("cert-dir", "/xe/pki", "where to read mounted certificates from")
+	certFname = flag.String("cert-fname", "cert.pem", "certificate filename")
+	keyFname  = flag.String("key-fname", "key.pem", "key filename")
+	proxyTo   = flag.String("proxy-to", "http://localhost:5000", "where to reverse proxy to")
+	slogLevel = flag.String("slog-level", "info", "logging level")
+)
+
+func main() {
+	flagenv.Parse()
+	flag.Parse()
+
+	internal.InitSlog(*slogLevel)
+
+	slog.Info("starting",
+		"bind", *bind,
+		"cert-dir", *certDir,
+		"cert-fname", *certFname,
+		"key-fname", *keyFname,
+		"proxy-to", *proxyTo,
+	)
+
+	cert := filepath.Join(*certDir, *certFname)
+	key := filepath.Join(*certDir, *keyFname)
+
+	st, err := os.Stat(cert)
+
+	if err != nil {
+		slog.Error("can't stat cert file", "certFname", cert)
+		os.Exit(1)
+	}
+
+	lastModified := st.ModTime()
+
+	go func(lm time.Time) {
+		t := time.NewTicker(time.Hour)
+		defer t.Stop()
+
+		for range t.C {
+			st, err := os.Stat(cert)
+			if err != nil {
+				slog.Error("can't stat file", "fname", cert, "err", err)
+				continue
+			}
+
+			if st.ModTime().After(lm) {
+				slog.Info("new cert detected", "oldTime", lm.Format(time.RFC3339), "newTime", st.ModTime().Format(time.RFC3339))
+				os.Exit(0)
+			}
+		}
+	}(lastModified)
+
+	u, err := url.Parse(*proxyTo)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	h := httputil.NewSingleHostReverseProxy(u)
+
+	if u.Scheme == "unix" {
+		slog.Info("using unix socket proxy")
+
+		h = &httputil.ReverseProxy{
+			Director: func(r *http.Request) {
+				r.URL.Scheme = "http"
+				r.URL.Host = r.Host
+
+				r.Header.Set("X-Forwarded-Proto", "https")
+				r.Header.Set("X-Forwarded-Scheme", "https")
+				r.Header.Set("X-Request-Id", uuid.NewString())
+				r.Header.Set("X-Scheme", "https")
+
+				remoteHost, remotePort, err := net.SplitHostPort(r.Host)
+				if err == nil {
+					r.Header.Set("X-Forwarded-Host", remoteHost)
+					r.Header.Set("X-Forwarded-Port", remotePort)
+				} else {
+					r.Header.Set("X-Forwarded-Host", r.Host)
+				}
+
+				host, _, err := net.SplitHostPort(r.RemoteAddr)
+				if err == nil {
+					r.Header.Set("X-Real-Ip", host)
+				}
+			},
+			Transport: &http.Transport{
+				DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+					return net.Dial("unix", strings.TrimPrefix(*proxyTo, "unix://"))
+				},
+			},
+		}
+	}
+
+	log.Fatal(
+		http.ListenAndServeTLS(
+			*bind,
+			cert,
+			key,
+			h,
+		),
+	)
+}

test/cmd/unixhttpd/main.go

@@ -0,0 +1,75 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"log/slog"
+	"net"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/TecharoHQ/anubis/internal"
+	"github.com/facebookgo/flagenv"
+)
+
+var (
+	dir        = flag.String("dir", ".", "directory to serve")
+	slogLevel  = flag.String("slog-level", "info", "logging level")
+	socketPath = flag.String("socket-path", "./unixhttpd.sock", "unix socket path to use")
+)
+
+func init() {
+	flag.Usage = func() {
+		fmt.Fprintf(os.Stderr, "Usage of %s:\n", filepath.Base(os.Args[0]))
+		fmt.Fprintf(os.Stderr, "  %s [--dir=.] [--socket-path=./unixhttpd.sock]\n\n", filepath.Base(os.Args[0]))
+		flag.PrintDefaults()
+		os.Exit(2)
+	}
+}
+
+func main() {
+	flagenv.Parse()
+	flag.Parse()
+
+	internal.InitSlog(*slogLevel)
+
+	if *dir == "" && *socketPath == "" {
+		flag.Usage()
+	}
+
+	slog.Info("starting up", "dir", *dir, "socketPath", *socketPath)
+
+	os.Remove(*socketPath)
+
+	mux := http.NewServeMux()
+
+	mux.HandleFunc("/reqmeta", func(w http.ResponseWriter, r *http.Request) {
+		contains := strings.Contains(r.Header.Get("Accept"), "text/html")
+
+		if contains {
+			w.Header().Add("Content-Type", "text/html")
+			fmt.Fprint(w, "<pre id=\"main\"><code>")
+		}
+
+		r.Write(w)
+
+		if contains {
+			fmt.Fprintln(w, "</pre></code>")
+		}
+	})
+
+	mux.Handle("/", http.FileServer(http.Dir(*dir)))
+
+	server := http.Server{
+		Handler: mux,
+	}
+
+	unixListener, err := net.Listen("unix", *socketPath)
+	if err != nil {
+		panic(err)
+	}
+	log.Fatal(server.Serve(unixListener))
+}

test/go.mod

@@ -0,0 +1,41 @@
+module github.com/TecharoHQ/anubis/test
+
+go 1.24.2
+
+replace github.com/TecharoHQ/anubis => ..
+
+require (
+	github.com/facebookgo/flagenv v0.0.0-20160425205200-fcd59fca7456
+	github.com/google/uuid v1.6.0
+)
+
+require (
+	github.com/TecharoHQ/anubis v1.16.0 // indirect
+	github.com/a-h/templ v0.3.857 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c // indirect
+	github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
+	github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
+	github.com/golang-jwt/jwt/v5 v5.2.2 // indirect
+	github.com/jsha/minica v1.1.0 // indirect
+	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
+	github.com/prometheus/client_golang v1.22.0 // indirect
+	github.com/prometheus/client_model v0.6.1 // indirect
+	github.com/prometheus/common v0.62.0 // indirect
+	github.com/prometheus/procfs v0.15.1 // indirect
+	github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a // indirect
+	github.com/yl2chen/cidranger v1.0.2 // indirect
+	golang.org/x/net v0.39.0 // indirect
+	golang.org/x/sys v0.32.0 // indirect
+	google.golang.org/protobuf v1.36.5 // indirect
+	k8s.io/apimachinery v0.32.3 // indirect
+	sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect
+	sigs.k8s.io/yaml v1.4.0 // indirect
+)
+
+tool (
+	github.com/TecharoHQ/anubis/cmd/anubis
+	github.com/jsha/minica
+)

test/go.sum

@@ -0,0 +1,55 @@
+github.com/a-h/templ v0.3.857 h1:6EqcJuGZW4OL+2iZ3MD+NnIcG7nGkaQeF2Zq5kf9ZGg=
+github.com/a-h/templ v0.3.857/go.mod h1:qhrhAkRFubE7khxLZHsBFHfX+gWwVNKbzKeF9GlPV4M=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c h1:8ISkoahWXwZR41ois5lSJBSVw4D0OV19Ht/JSTzvSv0=
+github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c/go.mod h1:Yg+htXGokKKdzcwhuNDwVvN+uBxDGXJ7G/VN1d8fa64=
+github.com/facebookgo/flagenv v0.0.0-20160425205200-fcd59fca7456 h1:CkmB2l68uhvRlwOTPrwnuitSxi/S3Cg4L5QYOcL9MBc=
+github.com/facebookgo/flagenv v0.0.0-20160425205200-fcd59fca7456/go.mod h1:zFhibDvPDWmtk4dAQ05sRobtyoffEHygEt3wSNuAzz8=
+github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 h1:JWuenKqqX8nojtoVVWjGfOF9635RETekkoH6Cc9SX0A=
+github.com/facebookgo/stack v0.0.0-20160209184415-751773369052/go.mod h1:UbMTZqLaRiH3MsBH8va0n7s1pQYcu3uTb8G4tygF4Zg=
+github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 h1:7HZCaLC5+BZpmbhCOZJ293Lz68O7PYrF2EzeiFMwCLk=
+github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
+github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
+github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/jsha/minica v1.1.0 h1:O2ZbzAN75w4RTB+5+HfjIEvY5nxRqDlwj3ZlLVG5JD8=
+github.com/jsha/minica v1.1.0/go.mod h1:dxC3wNmD+gU1ewXo/R8jB2ihB6wNpyXrG8aUk5Iuf/k=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
+github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
+github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
+github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
+github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
+github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
+github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
+github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
+github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
+github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a h1:iLcLb5Fwwz7g/DLK89F+uQBDeAhHhwdzB5fSlVdhGcM=
+github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a/go.mod h1:wozgYq9WEBQBaIJe4YZ0qTSFAMxmcwBhQH0fO0R34Z0=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/yl2chen/cidranger v1.0.2 h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=
+github.com/yl2chen/cidranger v1.0.2/go.mod h1:9U1yz7WPYDwf0vpNWFaeRh0bjwz5RVgRy/9UEQfHl0g=
+golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
+golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
+golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20=
+golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
+google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+k8s.io/apimachinery v0.32.3 h1:JmDuDarhDmA/Li7j3aPrwhpNBA94Nvk5zLeOge9HH1U=
+k8s.io/apimachinery v0.32.3/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 h1:/Rv+M11QRah1itp8VhT6HoVx1Ray9eB4DBr+K+/sCJ8=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3/go.mod h1:18nIHnGi6636UCz6m8i4DhaJ65T6EruyzmoQqI2BVDo=
+sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
+sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=

test/k8s/cert-manager/selfsigned-issuer.yaml

@@ -0,0 +1,6 @@
+apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: selfsigned
+spec:
+  selfSigned: {}

test/k8s/deps/cert-manager.yaml

@@ -0,0 +1,13 @@
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: cert-manager
+  namespace: kube-system
+spec:
+  repo: https://charts.jetstack.io
+  chart: cert-manager
+  targetNamespace: cert-manager
+  createNamespace: true
+  set:
+    installCRDs: "true"
+    "prometheus.enabled": "false"

test/nginx-external-auth/conf.d/default.conf

@@ -0,0 +1,25 @@
+server {
+  listen 80;
+  listen [::]:80;
+  server_name nginx.local.cetacean.club;
+
+  proxy_set_header X-Real-IP $remote_addr;
+  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+  location /.within.website/ {
+    proxy_pass http://localhost:8923;
+    auth_request off;
+  }
+
+  location @redirectToAnubis {
+    return 307 /.within.website/?redir=$scheme://$host$request_uri;
+    auth_request off;
+  }
+
+  location / {
+    auth_request /.within.website/x/cmd/anubis/api/check;
+    error_page 401 = @redirectToAnubis;
+    root /usr/share/nginx/html;
+    index index.html index.htm;
+  }
+}

test/nginx-external-auth/deployment.yaml

@@ -0,0 +1,50 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nginx-external-auth
+spec:
+  selector:
+    matchLabels:
+      app: nginx-external-auth
+  template:
+    metadata:
+      labels:
+        app: nginx-external-auth
+    spec:
+      volumes:
+      - name: config
+        configMap:
+          name: nginx-cfg
+      containers:
+      - name: www
+        image: nginx:alpine
+        resources:
+          limits:
+            memory: "128Mi"
+            cpu: "500m"
+          requests:
+            memory: "128Mi"
+            cpu: "500m"
+        ports:
+        - containerPort: 80
+        volumeMounts:
+        - name: config
+          mountPath: /etc/nginx/conf.d
+          readOnly: true
+      - name: anubis
+        image: ttl.sh/techaro/anubis-external-auth:latest
+        imagePullPolicy: Always
+        resources:
+          limits:
+            cpu: 500m
+            memory: 128Mi
+          requests:
+            cpu: 250m
+            memory: 128Mi
+        env:
+        - name: TARGET
+          value: " "
+        - name: REDIRECT_DOMAINS
+          value: nginx.local.cetacean.club
+
+

test/nginx-external-auth/ingress.yaml

@@ -0,0 +1,25 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: nginx-external-auth
+  labels:
+    name: nginx-external-auth
+  annotations:
+    cert-manager.io/cluster-issuer: "selfsigned"
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - nginx.local.cetacean.club
+    secretName: nginx-local-cetacean-club-public-tls
+  rules:
+  - host: nginx.local.cetacean.club
+    http:
+      paths:
+      - pathType: Prefix
+        path: "/"
+        backend:
+          service:
+            name: nginx-external-auth
+            port: 
+              name: http

test/nginx-external-auth/kustomization.yaml

@@ -0,0 +1,10 @@
+resources:
+  - deployment.yaml
+  - service.yaml
+  - ingress.yaml
+
+configMapGenerator:
+  - name: nginx-cfg
+    behavior: create
+    files:
+    - ./conf.d/default.conf

test/nginx-external-auth/service.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: nginx-external-auth
+spec:
+  selector:
+    app: nginx-external-auth
+  ports:
+  - name: http
+    protocol: TCP
+    port: 80
+    targetPort: 80
+  type: ClusterIP

test/nginx-external-auth/start.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# Build container image
+(
+  cd ../.. \
+  && npm ci \
+  && npm run container -- \
+      --docker-repo ttl.sh/techaro/anubis-external-auth \
+      --docker-tags ttl.sh/techaro/anubis-external-auth:latest
+)
+
+kubectl apply -k .
+echo "open https://nginx.local.cetacean.club, press control c when done"
+
+control_c() {
+  kubectl delete -k .
+  exit
+}
+trap control_c SIGINT
+
+sleep infinity

test/pki/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

test/shared/www/index.html

@@ -0,0 +1,17 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>Anubis works!</title>
+    <link rel="stylesheet" href="/.within.website/x/xess/xess.css"/>
+    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
+  </head>
+  <body id="top">
+    <main>
+      <h1>Anubis works!</h1>
+
+      <p>If you see this, everything has gone according to keikaku.</p>
+
+      <img height=128 src="/.within.website/x/cmd/anubis/static/img/happy.webp"/>
+    </main>
+  </body>
+</html>

test/unix-socket-xff/start.sh

@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# Remove lingering .sock files, relayd and unixhttpd will do that too but
+# measure twice, cut once.
+rm *.sock ||:
+
+# If the transient local TLS certificate doesn't exist, mint a new one
+if [ ! -f ../pki/relayd.local.cetacean.club/cert.pem ]; then
+  # Subshell to contain the directory change
+  (
+    cd ../pki \
+    && mkdir -p relayd.local.cetacean.club \
+    && \
+    # Try using https://github.com/FiloSottile/mkcert for better DevEx,
+    # but fall back to using https://github.com/jsha/minica in case
+    # you don't have that installed.
+    (
+      mkcert \
+        --cert-file ./relayd.local.cetacean.club/cert.pem \
+        --key-file ./relayd.local.cetacean.club/key.pem relayd.local.cetacean.club \
+      || go tool minica -domains relayd.local.cetacean.club
+    )
+  )
+fi
+
+# Build static assets
+(cd ../.. && npm ci && npm run assets)
+
+# Spawn three jobs:
+
+# HTTP daemon that listens over a unix socket (implicitly ./unixhttpd.sock)
+go run ../cmd/unixhttpd &
+
+# A copy of Anubis, specifically for the current Git checkout
+go tool anubis \
+  --bind=./anubis.sock \
+  --bind-network=unix \
+  --target=unix://$(pwd)/unixhttpd.sock &
+
+# A simple TLS terminator that forwards to Anubis, which will forward to
+# unixhttpd
+go run ../cmd/relayd \
+  --proxy-to=unix://./anubis.sock \
+  --cert-dir=../pki/relayd.local.cetacean.club &
+
+# When you press control c, kill all the child processes to clean things up
+trap 'echo signal received!; kill $(jobs -p); wait' SIGINT SIGTERM
+
+echo "open https://relayd.local.cetacean.club:3004/reqmeta"
+
+# Wait for all child processes to exit
+wait