test: add default-config-macro smoke test

This uses an AI generated python script to diff the contents of the bots
field of the default configuration file and the
data/meta/default-config.yaml file. It emits a patch showing what needs
to be changed.

Signed-off-by: Xe Iaso <me@xeiaso.net>

data/botPolicies.yaml

@@ -107,6 +107,7 @@ bots:
         - '"Sec-Fetch-Dest" in headers'
         - '"Sec-Fetch-Mode" in headers'
         - '"Sec-Fetch-Site" in headers'
+        - '"Upgrade-Insecure-Requests" in headers'
         - '"Accept-Encoding" in headers'
         - '( headers["Accept-Encoding"].contains("zstd") || headers["Accept-Encoding"].contains("br") )'
         - '"Accept-Language" in headers'
@@ -114,13 +115,6 @@ bots:
     weight:
       adjust: -10
 
-  # The Upgrade-Insecure-Requests header is typically sent by browsers, but not always
-  - name: upgrade-insecure-requests
-    expression: '"Upgrade-Insecure-Requests" in headers'
-    action: WEIGH
-    weight:
-      adjust: -2
-
   # Chrome should behave like Chrome
   - name: chrome-is-proper
     expression:

data/meta/default-config.yaml

@@ -91,6 +91,7 @@
       - '"Sec-Fetch-Dest" in headers'
       - '"Sec-Fetch-Mode" in headers'
       - '"Sec-Fetch-Site" in headers'
+      - '"Upgrade-Insecure-Requests" in headers'
       - '"Accept-Encoding" in headers'
       - '( headers["Accept-Encoding"].contains("zstd") || headers["Accept-Encoding"].contains("br") )'
       - '"Accept-Language" in headers'
@@ -98,13 +99,6 @@
   weight:
     adjust: -10
 
-# The Upgrade-Insecure-Requests header is typically sent by browsers, but not always
-- name: upgrade-insecure-requests
-  expression: '"Upgrade-Insecure-Requests" in headers'
-  action: WEIGH
-  weight:
-    adjust: -2
-
 # Chrome should behave like Chrome
 - name: chrome-is-proper
   expression:

docs/docs/CHANGELOG.md

@@ -22,7 +22,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Document missing environment variables in installation guide: `SLOG_LEVEL`, `COOKIE_PREFIX`, `FORCED_LANGUAGE`, and `TARGET_DISABLE_KEEPALIVE` ([#1086](https://github.com/TecharoHQ/anubis/pull/1086)).
 - Add validation warning when persistent storage is used without setting signing keys.
 - Fixed `robots2policy` to properly group consecutive user agents into `any:` instead of only processing the last one ([#925](https://github.com/TecharoHQ/anubis/pull/925)).
-- Make the `fast` algorithm prefer purejs when running in an insecure context.
 - Add the [`s3api` storage backend](./admin/policies.mdx#s3api) to allow Anubis to use S3 API compatible object storage as its storage backend.
 - Fix a "stutter" in the cookie name prefix so the auth cookie is named `techaro.lol-anubis-auth` instead of `techaro.lol-anubis-auth-auth`.
 - Make `cmd/containerbuild` support commas for separating elements of the `--docker-tags` argument as well as newlines.

docs/docs/admin/configuration/redirect-domains.mdx

@@ -32,7 +32,7 @@ sequenceDiagram
   participant Validation
   participant Evil Site
 
-  Hacker->>+User: Click on example.org with this solution
+  Hacker->>+User: Click on yoursite.com with this solution
   User->>+Validation: Here's a solution, send me to evilsite.com
   Validation->>+User: Here's a cookie, go to evilsite.com
   User->>+Evil Site: GET evilsite.com
@@ -46,14 +46,11 @@ Redirect domain not allowed
 
 ## Configuring allowed redirect domains
 
-By default, Anubis may redirect to any domain which could cause security issues in the unlikely case that an attacker passes a challenge for your browser and then tricks you into clicking a link to your domain.
-One can restrict the domains that Anubis can redirect to when passing a challenge by setting up `REDIRECT_DOMAINS` environment variable.
-If you need to set more than one domain, fill the environment variable with a comma-separated list of domain names.
-There is also glob matching support. You can pass `*.bugs.techaro.lol` to allow redirecting to anything ending with `.bugs.techaro.lol`. There is a limit of 4 wildcards.
+By default, Anubis will limit redirects to be on the same HTTP Host that Anubis is running on (EG: requests to yoursite.com cannot redirect outside of yoursite.com). If you need to set more than one domain, fill the `REDIRECT_DOMAINS` environment variable with a comma-separated list of domain names that Anubis should allow redirects to.
 
 :::note
 
-If you are hosting Anubis on a non-standard port (`https://example:com:8443`, `http://www.example.net:8080`, etc.), you must also include the port number here.
+These domains are _an exact string match_, they do not support wildcard matches.
 
 :::
 
@@ -63,7 +60,7 @@ If you are hosting Anubis on a non-standard port (`https://example:com:8443`, `h
 ```shell
 # anubis.env
 
-REDIRECT_DOMAINS="example.org,secretplans.example.org,*.test.example.org"
+REDIRECT_DOMAINS="yoursite.com,secretplans.yoursite.com"
 # ...
 ```
 
@@ -75,7 +72,7 @@ services:
   anubis-nginx:
     image: ghcr.io/techarohq/anubis:latest
     environment:
-      REDIRECT_DOMAINS: "example.org,secretplans.example.org,*.test.example.org"
+      REDIRECT_DOMAINS: "yoursite.com,secretplans.yoursite.com"
       # ...
 ```
 
@@ -89,7 +86,7 @@ Inside your Deployment, StatefulSet, or Pod:
   image: ghcr.io/techarohq/anubis:latest
   env:
     - name: REDIRECT_DOMAINS
-      value: "example.org,secretplans.example.org,*.test.example.org"
+      value: "yoursite.com,secretplans.yoursite.com"
     # ...
 ```
 

docs/docs/admin/installation.mdx

@@ -95,7 +95,7 @@ Anubis uses these environment variables for configuration:
 | `OVERLAY_FOLDER`               | unset                   | <EO /> If set, treat the given path as an [overlay folder](./botstopper.mdx#custom-images-and-css), allowing you to customize CSS, fonts, images, and add other assets to BotStopper deployments.                                                                                                                                                                                                                                                                                                                                              |
 | `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.mdx). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                                                                                                                                                                                                                                                                     |
 | `PUBLIC_URL`                   | unset                   | The externally accessible URL for this Anubis instance, used for constructing redirect URLs (e.g., for Traefik forwardAuth).                                                                                                                                                                                                                                                                                                                                                                                                                   |
-| `REDIRECT_DOMAINS`             | unset                   | Comma-separated list of domain names that Anubis should allow redirects to when passing a challenge. See [Redirect Domain Configuration](./configuration/redirect-domains) for more details.                                                                                                                                                                                                                                                                                          |
+| `REDIRECT_DOMAINS`             | unset                   | If set, restrict the domains that Anubis can redirect to when passing a challenge.<br/><br/>If this is unset, Anubis may redirect to any domain which could cause security issues in the unlikely case that an attacker passes a challenge for your browser and then tricks you into clicking a link to your domain.<br/><br/>Note that if you are hosting Anubis on a non-standard port (`https://example:com:8443`, `http://www.example.net:8080`, etc.), you must also include the port number here.                                        |
 | `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file.                                                                                                                                                                                                                                                       |
 | `SLOG_LEVEL`                   | `INFO`                  | The log level for structured logging. Valid values are `DEBUG`, `INFO`, `WARN`, and `ERROR`. Set to `DEBUG` to see all requests, evaluations, and detailed diagnostic information.                                                                                                                                                                                                                                                                                                                                                             |
 | `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                                                                                                                                                                                                                                                                      |

lib/anubis_test.go

@@ -194,7 +194,6 @@ func (u *userAgentRoundTripper) RoundTrip(req *http.Request) (*http.Response, er
 	// Only set if not already present
 	req = req.Clone(req.Context()) // avoid mutating original request
 	req.Header.Set("User-Agent", "Mozilla/5.0")
-	req.Header.Set("Accept-Encoding", "gzip")
 	return u.rt.RoundTrip(req)
 }
 

web/js/algorithms/fast.ts

@@ -18,12 +18,7 @@ export default function process(
 ): Promise<string> {
   console.debug("fast algo");
 
-  // Choose worker based on secure context.
-  // Use the WebCrypto worker if the page is a secure context; otherwise fall back to pure‑JS.
-  let workerMethod: "webcrypto" | "purejs" = "purejs";
-  if (window.isSecureContext) {
-    workerMethod = "webcrypto";
-  }
+  let workerMethod = window.crypto !== undefined ? "webcrypto" : "purejs";
 
   if (navigator.userAgent.includes("Firefox") || navigator.userAgent.includes("Goanna")) {
     console.log("Firefox detected, using pure-JS fallback");
@@ -87,4 +82,4 @@ export default function process(
       workers.push(worker);
     }
   });
-}
+}