Compare commits

...

15 Commits

Author SHA1 Message Date
Xe Iaso
617812a039 chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-06 02:29:54 +00:00
Xe Iaso
9ddbc69c0e docs(index): rewrite copy, add CELPHASE illustrations
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-06 02:00:50 +00:00
Xe Iaso
edb7ab1ae9 docs(index): start cleanup, remove proof of work from core branding
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-06 01:20:42 +00:00
Xe Iaso
7d0c58d1a8 fix: make ogtags and dnsbl use the Store instead of memory (#760)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:17:46 -04:00
Xe Iaso
e870ede120 docs(known-instances): add git.aya.so
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:41:36 +00:00
Xe Iaso
592d1e3dfc docs(known-instances): add Pluralpedia
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:40:32 +00:00
Xe Iaso
f6254b4b98 docs(installation): clarify BASE_PREFIX matches the /.within.website endpoints
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 13:47:02 +00:00
Lothar Serra Mari
d19026d693 docs(known-instances): Add Duke University, coinhoards.org (and myself) to known instances (#757)
* docs(known-instances): add Duke University to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* docs(known-instances): add fabulous.systems to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* docs(known-instances): add coinhoards.org to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* chore(spelling): exempt the known instances page

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Lothar Serra Mari <mail@serra.me>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-07-05 08:29:44 -04:00
Xe Iaso
7b72c790ab chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 12:29:19 +00:00
Xe Iaso
719a1409ca test(lib/store/bbolt): disable this test case for now
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 04:56:19 +00:00
Xe Iaso
890f21bf47 chore(devcontainer): move playwright to its own devcontainer service (#756)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 00:53:45 -04:00
Xe Iaso
93bfe910d8 docs(user/faq): clarify Anubis not being a cryptocurrency miner
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 22:49:39 +00:00
Xe Iaso
19d8de784b chore(docs/manifest): enable bbolt in an emptyDir
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 21:07:37 +00:00
Xe Iaso
dff2176beb feat(lib): use new challenge creation flow (#749)
* feat(decaymap): add Delete method

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(lib/challenge): refactor Validate to take ValidateInput

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib): implement store interface

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): all metapackage to import all store implementations

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(policy): import all store backends

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib): use new challenge creation flow

Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.

This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(decaymap): fix documentation typo

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(lib): fix SA4004

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/store): make generic storage interface test adaptor

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(decaymap): invert locking process for Delete

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): add bbolt store implementation

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: go mod tidy

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(devcontainer): adapt to docker compose, add valkey service

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): make challenges live for 30 minutes by default

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): implement valkey backend

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/store/valkey): disable tests if not using docker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/policy/config): ensure valkey stores can be loaded

Signed-off-by: Xe Iaso <me@xeiaso.net>

* Update metadata

check-spelling run (pull_request) for Xe/store-interface

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* chore(devcontainer): remove port forwards because vs code handles that for you

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(default-config): add a nudge to the storage backends section of the docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(docs): listen on 0.0.0.0 for dev container support

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(policy): document storage backends

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG and internal links

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(admin/policies): don't start a sentence with as

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: fixes found in review

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
2025-07-04 20:42:28 +00:00
SGHFan
506d8817d5 Update known-instances.md (#755)
Add eBird to instance list

Signed-off-by: SGHFan <62897482+SGHFan@users.noreply.github.com>
2025-07-04 19:54:26 +00:00
70 changed files with 1728 additions and 333 deletions

View File

@@ -3,10 +3,10 @@ FROM ghcr.io/xe/devcontainer-base/pre/go
WORKDIR /app
COPY go.mod go.sum package.json package-lock.json ./
RUN go install github.com/a-h/templ/cmd/templ \
&& npx --yes playwright@1.52.0 install --with-deps\
&& apt-get update \
&& apt-get -y install zstd brotli \
RUN apt-get update \
&& apt-get -y install zstd brotli redis \
&& mkdir -p /home/vscode/.local/share/fish \
&& chown -R vscode:vscode /home/vscode/.local/share/fish \
&& chown -R vscode:vscode /go
&& chown -R vscode:vscode /go
CMD ["/usr/bin/sleep", "infinity"]

View File

@@ -2,17 +2,13 @@
// README at: https://github.com/devcontainers/templates/tree/main/src/debian
{
"name": "Dev",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"build": {
"dockerfile": "./Dockerfile",
"context": "..",
"cacheFrom": [
"type=registry,ref=ghcr.io/techarohq/anubis/devcontainer"
]
},
"dockerComposeFile": ["./docker-compose.yaml"],
"service": "workspace",
"workspaceFolder": "/workspace/anubis",
"postStartCommand": "npm ci && go mod download",
"features": {
"ghcr.io/xe/devcontainer-features/ko:1.1.0": {}
"ghcr.io/xe/devcontainer-features/ko:1.1.0": {},
"ghcr.io/devcontainers/features/github-cli:1": {}
},
"initializeCommand": "mkdir -p ${localEnv:HOME}${localEnv:USERPROFILE}/.local/share/atuin",
"customizations": {
@@ -26,9 +22,5 @@
"redhat.vscode-yaml"
]
}
},
"forwardPorts": [
8923,
3000
]
}
}
}

View File

@@ -0,0 +1,26 @@
services:
playwright:
image: mcr.microsoft.com/playwright:v1.52.0-noble
init: true
network_mode: service:workspace
command:
- /bin/sh
- -c
- npx -y playwright@1.52.0 run-server --port 9001 --host 0.0.0.0
valkey:
image: valkey/valkey:8
pull_policy: always
# VS Code workspace service
workspace:
image: ghcr.io/techarohq/anubis/devcontainer
build:
context: ..
dockerfile: .devcontainer/Dockerfile
volumes:
- ../:/workspace/anubis:cached
environment:
VALKEY_URL: redis://valkey:6379/0
#entrypoint: ["/usr/bin/sleep", "infinity"]
user: vscode

View File

@@ -84,6 +84,7 @@
^\Q.github/workflows/spelling.yml\E$
^data/crawlers/
^docs/blog/tags\.yml$
^docs/docs/user/known-instances.md$
^docs/manifest/.*$
^docs/static/\.nojekyll$
^lib/policy/config/testdata/bad/unparseable\.json$

View File

@@ -16,12 +16,12 @@ aspirational
atuin
azuretools
badregexes
bbolt
bdba
berr
bingbot
bitcoin
Bitcoin
bitrate
blogging
Bluesky
blueskybot
boi
@@ -46,11 +46,13 @@ cgr
chainguard
chall
challengemozilla
challengetest
checkpath
checkresult
chibi
cidranger
ckie
ckies
cloudflare
Codespaces
confd
@@ -67,6 +69,7 @@ DDOS
Debian
debrpm
decaymap
devcontainers
Diffbot
discordapp
discordbot
@@ -127,7 +130,6 @@ Hashcash
hashrate
headermap
healthcheck
hebis
hec
hmc
hostable
@@ -232,6 +234,7 @@ qwantbot
rac
rawler
rcvar
rdb
redhat
redir
redirectscheme
@@ -246,7 +249,6 @@ RUnlock
runtimedir
sas
sasl
Scumm
searchbot
searx
sebest
@@ -259,10 +261,8 @@ shellcheck
Sidetrade
simprint
sitemap
skopeo
sls
sni
Sourceware
Spambot
sparkline
spyderbot
@@ -270,6 +270,7 @@ srv
stackoverflow
startprecmd
stoppostcmd
storetest
subgrid
subr
subrequest
@@ -291,6 +292,8 @@ Tik
Timpibot
traefik
uberspace
Unbreak
unbreakdocker
unifiedjs
unixhttpd
unmarshal
@@ -298,7 +301,7 @@ unparseable
uuidgen
uvx
UXP
Valkey
valkey
Varis
Velen
vendored

View File

@@ -1,47 +0,0 @@
name: Dev container prebuild
on:
push:
branches: ["main"]
tags: ["v*.*.*"]
jobs:
devcontainer:
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-tags: true
fetch-depth: 0
persist-credentials: false
- name: Set up QEMU
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: latest
- run: |
sudo apt-get update
sudo apt-get -y install skopeo
- name: Log into registry
if: github.event_name != 'pull_request'
uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3.0.0
with:
registry: ghcr.io
username: techarohq
password: ${{ secrets.GITHUB_TOKEN }}
- name: Pre-build dev container image
uses: devcontainers/ci@8bf61b26e9c3a98f69cb6ce2f88d24ff59b785c6 # v0.3.1900000417
with:
imageName: ghcr.io/techarohq/anubis/devcontainer
cacheFrom: ghcr.io/techarohq/anubis/devcontainer
push: always
platform: linux/amd64,linux/arm64

View File

@@ -231,20 +231,6 @@ func makeReverseProxy(target string, targetSNI string, targetHost string, insecu
return rp, nil
}
func startDecayMapCleanup(ctx context.Context, s *libanubis.Server) {
ticker := time.NewTicker(1 * time.Hour)
defer ticker.Stop()
for {
select {
case <-ticker.C:
s.CleanupDecayMap()
case <-ctx.Done():
return
}
}
}
func main() {
flagenv.Parse()
flag.Parse()
@@ -421,7 +407,6 @@ func main() {
wg.Add(1)
go metricsServer(ctx, wg.Done)
}
go startDecayMapCleanup(ctx, s)
var h http.Handler
h = s

View File

@@ -145,6 +145,14 @@ status_codes:
CHALLENGE: 200
DENY: 200
# Anubis can store temporary data in one of a few backends. See the storage
# backends section of the docs for more information:
#
# https://anubis.techaro.lol/docs/admin/policies#storage-backends
store:
backend: memory
parameters: {}
# The weight thresholds for when to trigger individual challenges. Any
# CHALLENGE will take precedence over this.
#

View File

@@ -48,6 +48,26 @@ func (m *Impl[K, V]) expire(key K) bool {
return true
}
// Delete a value from the DecayMap by key.
//
// If the value does not exist, return false. Return true after
// deletion.
func (m *Impl[K, V]) Delete(key K) bool {
m.lock.RLock()
_, ok := m.data[key]
m.lock.RUnlock()
if !ok {
return false
}
m.lock.Lock()
delete(m.data, key)
m.lock.Unlock()
return true
}
// Get gets a value from the DecayMap by key.
//
// If a value has expired, forcibly delete it if it was not updated.

View File

@@ -22,6 +22,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Add option for custom cookie prefix ([#732](https://github.com/TecharoHQ/anubis/pull/732))
- Add translation for German language ([#741](https://github.com/TecharoHQ/anubis/pull/741))
- Remove the "Success" interstitial after a proof of work challenge is concluded.
- Anubis now has the concept of [storage backends](./admin/policies.mdx#storage-backends). These allow you to change how Anubis stores temporary data (in memory, on the disk, or in Valkey). If you run Anubis in an environment where you have a low amount of memory available for Anubis (eg: less than 64 megabytes), be sure to configure the [`bbolt`](./admin/policies.mdx#bbolt) storage backend.
- The challenge issuance and validation process has been rewritten from scratch. Instead of generating challenge strings from request metadata (under the assumption that the values being compared against are stable), Anubis now generates random data for each challenge. This data is stored in the active [storage backend](./admin/policies.mdx#storage-backends) for up to 30 minutes. Fixes [#564](https://github.com/TecharoHQ/anubis/issues/564), [#746](https://github.com/TecharoHQ/anubis/issues/746), and other similar instances of this issue.
- Make the [Open Graph](./admin/configuration/open-graph.mdx) subsystem and DNSBL subsystem use [storage backends](./admin/policies.mdx#storage-backends) instead of storing everything in memory by default.
- Add option for forcing a specific language ([#742](https://github.com/TecharoHQ/anubis/pull/742))
- Add translation for Turkish language ([#751](https://github.com/TecharoHQ/anubis/pull/751))
- Allow [Common Crawl](https://commoncrawl.org/) by default so scrapers have less incentive to scrape

View File

@@ -60,14 +60,14 @@ Anubis uses these environment variables for configuration:
| Environment Variable | Default value | Explanation |
| :----------------------------- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `BASE_PREFIX` | unset | If set, adds a global prefix to all Anubis endpoints. For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes. |
| `BASE_PREFIX` | unset | If set, adds a global prefix to all Anubis endpoints (everything starting with `/.within.website/x/anubis/`). For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes. |
| `BIND` | `:8923` | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock` |
| `BIND_NETWORK` | `tcp` | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports. |
| `COOKIE_DOMAIN` | unset | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See this [stackoverflow explanation of cookies](https://stackoverflow.com/a/1063760) for more information.<br/><br/>Note that unlike `REDIRECT_DOMAINS`, you should never include a port number in this variable. |
| `COOKIE_DYNAMIC_DOMAIN` | false | If set to true, automatically set cookie domain fields based on the hostname of the request. EG: if you are making a request to `anubis.techaro.lol`, the Anubis cookie will be valid for any subdomain of `techaro.lol`. |
| `COOKIE_EXPIRATION_TIME` | `168h` | The amount of time the authorization cookie is valid for. |
| `COOKIE_PARTITIONED` | `false` | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe. |
| `COOKIE_SECURE` | `true` | If set to `true`, enables the [Secure flag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies), meaning that the cookies will only be transmitted over HTTPS. If Anubis is used in an unsecure context (plain HTTP), this will be need to be set to false |
| `COOKIE_SECURE` | `true` | If set to `true`, enables the [Secure flag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies), meaning that the cookies will only be transmitted over HTTPS. If Anubis is used in an unsecure context (plain HTTP), this will be need to be set to false |
| `DIFFICULTY` | `4` | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses. |
| `ED25519_PRIVATE_KEY_HEX` | unset | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details. |
| `ED25519_PRIVATE_KEY_HEX_FILE` | unset | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set. |

View File

@@ -237,6 +237,115 @@ remote_addresses:
Anubis has support for showing imprint / impressum information. This is defined in the `impressum` block of your configuration. See [Imprint / Impressum configuration](./configuration/impressum.mdx) for more information.
## Storage backends
Anubis needs to store temporary data in order to determine if a user is legitimate or not. Administrators should choose a storage backend based on their infrastructure needs. Each backend has its own advantages and disadvantages.
Anubis offers the following storage backends:
- [`memory`](#memory) -- A simple in-memory hashmap
- [`bbolt`](#bbolt) -- An on-disk key/value store backed by [bbolt](https://github.com/etcd-io/bbolt), an embedded key/value database for Go programs
- [`valkey`](#valkey) -- A remote in-memory key/value database backed by [Valkey](https://valkey.io/) (or another database compatible with the [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/) protocol)
If no storage backend is set in the policy file, Anubis will use the [`memory`](#memory) backend by default. This is equivalent to the following in the policy file:
```yaml
store:
backend: memory
parameters: {}
```
### `memory`
The memory backend is an in-memory cache. This backend works best if you don't use multiple instances of Anubis or don't have mutable storage in the environment you're running Anubis in.
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | ✅ Yes |
| Does your service get a lot of traffic? | 🚫 No |
| Do you want to store data persistently when Anubis restarts? | 🚫 No |
| Do you run Anubis without mutable filesystem storage? | ✅ Yes |
The biggest downside is that there is not currently a limit to how much data can be stored in memory. This will be addressed at a later time.
#### Configuration
The memory backend does not require any configuration to use.
### `bbolt`
An on-disk storage layer powered by [bbolt](https://github.com/etcd-io/bbolt), a high performance embedded key/value database used by containerd, etcd, Kubernetes, and NATS. This backend works best if you're running Anubis on a single host and get a lot of traffic.
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | ✅ Yes |
| Does your service get a lot of traffic? | ✅ Yes |
| Do you want to store data persistently when Anubis restarts? | ✅ Yes |
| Do you run Anubis without mutable filesystem storage? | 🚫 No |
When Anubis opens a bbolt database, it takes an exclusive lock on that database. Other instances of Anubis or other tools cannot view the bbolt database while it is locked by another instance of Anubis. If you run multiple instances of Anubis for different services, give each its own `bbolt` configuration.
#### Configuration
The `bbolt` backend takes the following configuration options:
| Name | Type | Example | Description |
| :------- | :----- | :----------------- | :-------------------------------------------------------------------------------------------------------------------------------- |
| `bucket` | string | `anubis` | The bbolt bucket that Anubis should place all its data into. If this is not set, then Anubis will default to the bucket `anubis`. |
| `path` | path | `/data/anubis.bdb` | The filesystem path for the Anubis bbolt database. Anubis requires write access to the folder containing the bbolt database. |
Example:
If you have persistent storage mounted to `/data`, then your store configuration could look like this:
```yaml
store:
backend: bbolt
parameters:
path: /data/anubis.bdb
```
### `valkey`
[Valkey](https://valkey.io/) is an in-memory key/value store that clients access over the network. This allows multiple instances of Anubis to share information and does not require each instance of Anubis to have persistent filesystem storage.
:::note
You can also use [Redis](http://redis.io/) with Anubis.
:::
This backend is ideal if you are running multiple instances of Anubis in a worker pool (eg: Kubernetes Deployments with a copy of Anubis in each Pod).
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | 🚫 No |
| Does your service get a lot of traffic? | ✅ Yes |
| Do you want to store data persistently when Anubis restarts? | ✅ Yes |
| Do you run Anubis without mutable filesystem storage? | ✅ Yes |
| Do you have Redis or Valkey installed? | ✅ Yes |
#### Configuration
The `valkey` backend takes the following configuration options:
| Name | Type | Example | Description |
| :---- | :----- | :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | string | `redis://valkey:6379/0` | The URL for the instance of Redis or Valkey that Anubis should store data in. This is in the same format as `REDIS_URL` in many cloud providers. |
Example:
If you have an instance of Valkey running with the hostname `valkey.int.techaro.lol`, then your store configuration could look like this:
```yaml
store:
backend: valkey
parameters:
url: "redis://valkey.int.techaro.lol:6379/0"
```
This would have the Valkey client connect to host `valkey.int.techaro.lol` on port `6379` with database `0` (the default database).
## Risk calculation for downstream services
In case your service needs it for risk calculation reasons, Anubis exposes information about the rules that any requests match using a few headers:

View File

@@ -2,7 +2,7 @@
title: Why does Anubis use Proof-of-Work?
---
Anubis uses a [proof of work](https://en.wikipedia.org/wiki/Proof_of_work) in order to validate that clients are genuine. The reason Anubis does this was inspired by [Hashcash](https://en.wikipedia.org/wiki/Hashcash), a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation. This is also how Bitcoin's consensus algorithm works.
Anubis uses [proof of work](https://en.wikipedia.org/wiki/Proof_of_work) in order to validate that clients are genuine. The reason Anubis does this was inspired by [Hashcash](https://en.wikipedia.org/wiki/Hashcash), a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation. This is also how Bitcoin's consensus algorithm works.
## How Anubis' proof of work scheme works
@@ -21,16 +21,3 @@ const hash = await sha256(`${challenge}${nonce}`);
In order to pass a challenge, the `hash` has to have the right number of leading zeros (the "difficulty"). When a client requests to pass the challenge, they include the nonce they used. The server then only has to do one sha256 operation: the one that confirms that the challenge (generated from request metadata) and the nonce (provided by the client) match the difficulty number of leading zeroes.
Ultimately, this is a hack whose real purpose is to give a "good enough" placeholder solution so that more time can be spent on fingerprinting and identifying headless browsers (EG via how they do font rendering) so that the challenge proof of work page doesn't need to be presented to known legitimate users.
## Challenge format
Anubis generates challenges based on browser metadata, including but not limited to the following:
- The contents of your [`Accept-Language` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language)
- The IP address of your client
- Your browser's [`User-Agent` string](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent)
- The date of the current week, rooted on Sundays
- Anubis' ed25519 public signing key for [JSON web tokens](https://jwt.io/) (JWTs)
- The challenge difficulty
This is intended to be a random value that is difficult for attackers to forge and guess, but also deterministic enough that it will naturally reset itself.

View File

@@ -21,8 +21,4 @@ If you use a browser extension such as [JShelter](https://jshelter.org/), you wi
## Does Anubis mine Bitcoin?
No. Anubis does not mine Bitcoin.
In order to mine bitcoin, you need to download a copy of the blockchain (so you have the state required to do mining) and also broadcast your mined blocks to the network should you reach a hash with the right number of leading zeroes. You also need to continuously read for newly broadcasted transactions so you can batch them into a block. This requires gigabytes of data to be transferred from the server to the client.
Anubis transfers two digit numbers of kilobytes from the server to the client (which you can independently verify with your browser's Developer Tools feature). This is orders of magnitude below what is required to mine Bitcoin.
No. Anubis does not mine Bitcoin or any other cryptocurrency.

View File

@@ -45,6 +45,11 @@ This page contains a non-exhaustive list with all websites using Anubis.
- https://gitlab.postmarketos.org/
- https://wiki.koha-community.org/
- https://extensions.typo3.org/
- https://ebird.org/
- https://fabulous.systems/
- https://coinhoards.org/
- https://pluralpedia.org/
- https://git.aya.so/
- <details>
<summary>FreeCAD</summary>
- https://forum.freecad.org/
@@ -82,3 +87,10 @@ This page contains a non-exhaustive list with all websites using Anubis.
- https://karla.hds.hebis.de/
- and many more (see https://www.hebis.de/dienste/hebis-discovery-system/)
</details>
- <details>
<summary>Duke University</summary>
- https://repository.duke.edu/
- https://archives.lib.duke.edu/
- https://find.library.duke.edu/
- https://nicholas.duke.edu/
</details>

View File

@@ -6,7 +6,7 @@ import type * as Preset from '@docusaurus/preset-classic';
const config: Config = {
title: 'Anubis',
tagline: 'Weigh the soul of incoming HTTP requests using proof-of-work to stop AI crawlers',
tagline: 'Weigh the soul of incoming HTTP requests to protect your website!',
favicon: 'img/favicon.ico',
// Set the production url of your site here
@@ -40,27 +40,20 @@ const config: Config = {
[
'classic',
{
docs: {
sidebarPath: './sidebars.ts',
// Please change this to your repo.
// Remove this to remove the "edit this page" links.
editUrl:
'https://github.com/TecharoHQ/anubis/tree/main/docs/',
},
blog: {
showReadingTime: true,
feedOptions: {
type: ['rss', 'atom', "json"],
xslt: true,
},
// Please change this to your repo.
// Remove this to remove the "edit this page" links.
editUrl:
'https://github.com/facebook/docusaurus/tree/main/packages/create-docusaurus/templates/shared/',
// Useful options to enforce blogging best practices
editUrl: 'https://github.com/TecharoHQ/anubis/tree/main/docs/',
onInlineTags: 'warn',
onInlineAuthors: 'warn',
onUntruncatedBlogPosts: 'warn',
onUntruncatedBlogPosts: 'throw',
},
docs: {
sidebarPath: './sidebars.ts',
editUrl: 'https://github.com/TecharoHQ/anubis/tree/main/docs/',
},
theme: {
customCss: './src/css/custom.css',
@@ -74,7 +67,7 @@ const config: Config = {
respectPrefersColorScheme: true,
},
// Replace with your project's social card
image: 'img/docusaurus-social-card.jpg',
image: 'img/social-card.jpg',
navbar: {
title: 'Anubis',
logo: {
@@ -82,23 +75,28 @@ const config: Config = {
src: 'img/favicon.webp',
},
items: [
{ to: '/blog', label: 'Blog', position: 'left' },
{
type: 'docSidebar',
sidebarId: 'tutorialSidebar',
position: 'left',
label: 'Docs',
},
{ to: '/blog', label: 'Blog', position: 'left' },
{
href: 'https://github.com/sponsors/Xe',
label: "Sponsorship",
position: 'left'
to: '/docs/admin/botstopper',
label: "Unbranded Version",
position: "left"
},
{
href: 'https://github.com/TecharoHQ/anubis',
label: 'GitHub',
position: 'right',
},
{
href: 'https://github.com/sponsors/Xe',
label: "Sponsor the Project",
position: 'right'
},
],
},
footer: {

View File

@@ -127,3 +127,8 @@ impressum:
status_codes:
CHALLENGE: 200
DENY: 200
store:
backend: bbolt
parameters:
path: /xe/data/anubis/data.bdb

View File

@@ -15,6 +15,8 @@ spec:
- name: anubis
configMap:
name: anubis-cfg
- name: temporary-data
emptyDir: {}
containers:
- name: anubis-docs
image: ghcr.io/techarohq/anubis/docs:main
@@ -51,6 +53,8 @@ spec:
volumeMounts:
- name: anubis
mountPath: /xe/cfg/anubis
- name: temporary-data
mountPath: /xe/data/anubis
resources:
limits:
cpu: 500m

View File

@@ -4,7 +4,7 @@
"private": true,
"scripts": {
"docusaurus": "docusaurus",
"start": "docusaurus start",
"start": "docusaurus start --host 0.0.0.0",
"build": "docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "echo 'use CI' && exit 1",
@@ -45,4 +45,4 @@
"engines": {
"node": ">=18.0"
}
}
}

View File

@@ -5,49 +5,50 @@ import styles from "./styles.module.css";
type FeatureItem = {
title: string;
Svg: React.ComponentType<React.ComponentProps<"svg">>;
imageURL: string;
description: ReactNode;
};
const FeatureList: FeatureItem[] = [
{
title: "Easy to Use",
Svg: require("@site/static/img/undraw_docusaurus_mountain.svg").default,
imageURL: require("@site/static/img/anubis/happy.webp").default,
description: (
<>
Anubis is easy to set up, lightweight, and helps get rid of the lowest
hanging fruit so you can sleep at night.
Anubis sits in the background and weighs the risk of incoming requests.
If it asks a client to complete a challenge, no user interaction is
required.
</>
),
},
{
title: "Lightweight",
Svg: require("@site/static/img/undraw_docusaurus_tree.svg").default,
imageURL: require("@site/static/img/anubis/pensive.webp").default,
description: (
<>
Anubis is efficient and as lightweight as possible, blocking the worst
of the bots on the internet and makes it easy to protect what you host
online.
Anubis is so lightweight you'll forget it's there until you look at your
hosting bill. On average it uses less than 128 MB of ram.
</>
),
},
{
title: "Multi-threaded",
Svg: require("@site/static/img/undraw_docusaurus_react.svg").default,
title: "Block the scrapers",
imageURL: require("@site/static/img/anubis/reject.webp").default,
description: (
<>
Anubis uses a multi-threaded proof of work check to ensure that users
browsers are up to date and support modern standards.
Anubis uses a combination of heuristics to identify and block bots
before they take your website down. You can customize the rules with{" "}
<a href="/docs/admin/policies">your own policies</a>.
</>
),
},
];
function Feature({ title, Svg, description }: FeatureItem) {
function Feature({ title, description, imageURL }: FeatureItem) {
return (
<div className={clsx("col col--4")}>
<div className="text--center">
<Svg className={styles.featureSvg} role="img" />
<img src={imageURL} className={styles.featureSvg} role="img" />
</div>
<div className="text--center padding-horiz--md">
<Heading as="h3">{title}</Heading>

View File

@@ -31,19 +31,12 @@ export default function Home(): ReactNode {
const { siteConfig } = useDocusaurusContext();
return (
<Layout
title={`Anubis: self hostable scraper defense software`}
description="Weigh the soul of incoming HTTP requests using proof-of-work to stop AI crawlers"
title={`Anubis: Web AI Firewall Utility`}
description="Weigh the soul of incoming HTTP requests to protect your website!"
>
<HomepageHeader />
<main>
<HomepageFeatures />
<center>
<p>
This is all placeholder text. It will be fixed. Give me time. I am
one person and my project has unexpectedly gone viral.
</p>
</center>
</main>
</Layout>
);

BIN
docs/static/img/anubis/happy.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

BIN
docs/static/img/anubis/pensive.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

BIN
docs/static/img/anubis/reject.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

BIN
docs/static/img/social-card.jpg vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 881 KiB

44
go.mod
View File

@@ -10,13 +10,16 @@ require (
github.com/gaissmai/bart v0.20.4
github.com/golang-jwt/jwt/v5 v5.2.2
github.com/google/cel-go v0.25.0
github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus v1.1.0
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2
github.com/joho/godotenv v1.5.1
github.com/nicksnyder/go-i18n/v2 v2.6.0
github.com/playwright-community/playwright-go v0.5200.0
github.com/prometheus/client_golang v1.22.0
github.com/redis/go-redis/v9 v9.11.0
github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a
go.etcd.io/bbolt v1.4.2
golang.org/x/net v0.41.0
golang.org/x/text v0.26.0
google.golang.org/grpc v1.73.0
@@ -31,6 +34,7 @@ require (
cel.dev/expr v0.23.1 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/AlekSi/pointer v1.2.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/BurntSushi/toml v1.5.0 // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.3.1 // indirect
@@ -49,31 +53,45 @@ require (
github.com/cli/browser v1.3.0 // indirect
github.com/cli/go-gh v0.1.0 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.6 // indirect
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/deckarep/golang-set/v2 v2.8.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/dlclark/regexp2 v1.11.4 // indirect
github.com/docker/docker v28.0.1+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c // indirect
github.com/ebitengine/purego v0.8.2 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51 // indirect
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 // indirect
github.com/fatih/color v1.17.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-git/go-git/v5 v5.14.0 // indirect
github.com/go-jose/go-jose/v3 v3.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-yaml v1.12.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/go-github/v70 v70.0.0 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/pprof v0.0.0-20230207041349-798e818bf904 // indirect
github.com/google/rpmpack v0.6.1-0.20250405124433-758cc6896cbc // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/goreleaser/chglog v0.7.0 // indirect
github.com/goreleaser/fileglob v1.3.0 // indirect
github.com/goreleaser/nfpm/v2 v2.42.1 // indirect
@@ -83,33 +101,57 @@ require (
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.5.0 // indirect
github.com/moby/sys/user v0.1.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/natefinch/atomic v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shirou/gopsutil/v4 v4.25.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
github.com/spf13/afero v1.14.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/stretchr/testify v1.10.0 // indirect
github.com/suzuki-shunsuke/logrus-error v0.1.4 // indirect
github.com/suzuki-shunsuke/pinact v1.6.0 // indirect
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4 // indirect
github.com/testcontainers/testcontainers-go v0.37.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/ulikunitz/xz v0.5.12 // indirect
github.com/urfave/cli/v2 v2.27.6 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
gitlab.com/digitalxero/go-conventional-commit v1.0.7 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.35.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/otel/trace v1.35.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v3 v3.0.3 // indirect
golang.org/x/crypto v0.39.0 // indirect

91
go.sum
View File

@@ -8,6 +8,8 @@ dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
github.com/AlekSi/pointer v1.2.0 h1:glcy/gc4h8HnG2Z3ZECSzZ1IX1x2JxRVuDzaJwQE0+w=
github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tSNSBle0=
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/BurntSushi/toml v1.5.0 h1:W5quZX/G/csjUnuI8SUYlsHs9M38FC7znL0lIO+DvMg=
github.com/BurntSushi/toml v1.5.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/DataDog/zstd v1.5.5 h1:oWf5W7GtOLgp6bciQYDmhHHjdhYkALu6S/5Ni9ZgSvQ=
@@ -50,6 +52,10 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb h1:m935MPodAbYS46DG4pJSv7WO+VECIWUQ7OJYSoTrMh4=
github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb/go.mod h1:PkYb9DJNAwrSvRx5DYA+gUcOIgTGVMNkfSCbZM8cWpI=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/caarlos0/testfs v0.4.4 h1:3PHvzHi5Lt+g332CiShwS8ogTgS3HjrmzZxCm6JCDr8=
github.com/caarlos0/testfs v0.4.4/go.mod h1:bRN55zgG4XCUVVHZCeU+/Tz1Q6AxEJOEJTliBy+1DMk=
github.com/cavaliergopher/cpio v1.0.1 h1:KQFSeKmZhv0cr+kawA3a0xTQCU4QxXF1vhU7P7av2KM=
@@ -69,6 +75,12 @@ github.com/cli/safeexec v1.0.0/go.mod h1:Z/D4tTN8Vs5gXYHDCbaM1S/anmEDnJb1iW0+EJ5
github.com/cli/shurcooL-graphql v0.0.1/go.mod h1:U7gCSuMZP/Qy7kbqkk5PrqXEeDgtfG5K+W+u8weorps=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/cpuguy83/go-md2man/v2 v2.0.6 h1:XJtiaUW6dEEqVuZiMTn1ldk455QWwEIsMIJlo5vtkx0=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
@@ -81,10 +93,22 @@ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/deckarep/golang-set/v2 v2.8.0 h1:swm0rlPCmdWn9mESxKOjWk8hXSqoxOp+ZlfuyaAdFlQ=
github.com/deckarep/golang-set/v2 v2.8.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/dlclark/regexp2 v1.11.4 h1:rPYF9/LECdNymJufQKmri9gV604RvvABwgOA8un7yAo=
github.com/dlclark/regexp2 v1.11.4/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/docker/docker v28.0.1+incompatible h1:FCHjSRdXhNRFjlHMTv4jUNlIBbTeRjrWfeFuJp7jpo0=
github.com/docker/docker v28.0.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c h1:mxWGS0YyquJ/ikZOjSrRjjFIbUqIP9ojyYQ+QZTU3Rg=
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c/go.mod h1:MxLav0peU43GgvwVgNbLAj1s/bSGboKkhuULvq/7hx4=
github.com/ebitengine/purego v0.8.2 h1:jPPGWs2sZ1UgOSgD2bClL0MJIqu58nOmIcBuXr62z1I=
github.com/ebitengine/purego v0.8.2/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE=
github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
@@ -101,6 +125,8 @@ github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGE
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/fatih/color v1.17.0 h1:GlRw1BRJxkpqUCBKzKOw098ed57fEsKeNjpTe3cSjK4=
github.com/fatih/color v1.17.0/go.mod h1:YZ7TlrGPkiz6ku9fK3TLD/pl3CpsiFyu8N92HLgmosI=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
@@ -119,10 +145,13 @@ github.com/go-git/go-git/v5 v5.14.0 h1:/MD3lCrGjCen5WfEAzKg00MJJffKhC8gzS80ycmCi
github.com/go-git/go-git/v5 v5.14.0/go.mod h1:Z5Xhoia5PcWA3NF8vRLURn9E5FRhSl7dGj9ItW3Wk5k=
github.com/go-jose/go-jose/v3 v3.0.4 h1:Wp5HA7bLQcKnf6YYao/4kpRpVMp/yf6+pJKV8WFSaNY=
github.com/go-jose/go-jose/v3 v3.0.4/go.mod h1:5b+7YgP7ZICgJDBdfjZaIt+H/9L9T/YQrVfLAMboGkQ=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
@@ -142,6 +171,8 @@ github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJA
github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
github.com/goccy/go-yaml v1.12.0 h1:/1WHjnMsI1dlIBQutrvSMGZRQufVO3asrHfTwfACoPM=
github.com/goccy/go-yaml v1.12.0/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
@@ -153,6 +184,7 @@ github.com/google/cel-go v0.25.0/go.mod h1:hjEb6r5SuOSlhCHmFoLzu8HGCERvIsDAbxDAy
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786 h1:rcv+Ippz6RAtvaGgKxc+8FQIpxHgsF+HBzPyYL2cyVU=
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786/go.mod h1:apVn/GCasLZUVpAJ6oWAuyP7Ne7CEsQbTnc0plM3m+o=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
@@ -196,6 +228,8 @@ github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d h1:RnWZeH8N8KXfbwMTex/KKMYMj0FJRCF6tQubUuQ02GM=
github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d/go.mod h1:phT/jsRPBAEqjAibu1BurrabCBNTYiVI+zbmyCZJY6Q=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
@@ -213,6 +247,10 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/matryer/is v1.4.0 h1:sosSmIWwkYITGrxZ25ULNDeKiMNzFSr4V/eqBQP0PeE=
github.com/matryer/is v1.4.0/go.mod h1:8I/i5uYgLzgsgEloJE1U6xx5HkBQpAZvepWuujKwMRU=
github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
@@ -232,6 +270,20 @@ github.com/mitchellh/go-ps v1.0.0 h1:i6ampVEEF4wQFF+bkYfwYgY+F/uYJDktmvLPf7qIgjc
github.com/mitchellh/go-ps v1.0.0/go.mod h1:J4lOc8z8yJs6vUwklHw2XEIiT4z4C40KtWVN3nvg8Pg=
github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc=
github.com/moby/sys/sequential v0.5.0/go.mod h1:tH2cOOs5V9MlPiXcQzRC+eEyab644PWKGRYaaV5ZZlo=
github.com/moby/sys/user v0.1.0 h1:WmZ93f5Ux6het5iituh9x2zAG7NFY9Aqi49jjE1PaQg=
github.com/moby/sys/user v0.1.0/go.mod h1:fKJhFOnsCN6xZ5gSfbM6zaHGgDJMrqt9/reuj4T7MmU=
github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/natefinch/atomic v1.0.1 h1:ZPYKxkqQOx3KZ+RsbnP/YsgvxWQPGxjC0oBt2AhwV0A=
@@ -241,6 +293,10 @@ github.com/nicksnyder/go-i18n/v2 v2.6.0 h1:C/m2NNWNiTB6SK4Ao8df5EWm3JETSTIGNXBpM
github.com/nicksnyder/go-i18n/v2 v2.6.0/go.mod h1:88sRqr0C6OPyJn0/KRNaEz1uWorjxIKP7rUUcvycecE=
github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/pjbgf/sha1cd v0.3.2 h1:a9wb0bp1oC2TGwStyn0Umc/IGKQnEgF0vVaZ8QF8eo4=
github.com/pjbgf/sha1cd v0.3.2/go.mod h1:zQWigSxVmsHEZow5qaLtPYxpcKMMQpa09ixqBxuCS6A=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -250,6 +306,8 @@ github.com/playwright-community/playwright-go v0.5200.0/go.mod h1:UnnyQZaqUOO5yw
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
@@ -258,6 +316,8 @@ github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ
github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/redis/go-redis/v9 v9.11.0 h1:E3S08Gl/nJNn5vkxd2i78wZxWAPNZgUNTp8WIJUAiIs=
github.com/redis/go-redis/v9 v9.11.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
@@ -269,6 +329,8 @@ github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a h1:iLcLb5Fwwz7g/DLK89F+
github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a/go.mod h1:wozgYq9WEBQBaIJe4YZ0qTSFAMxmcwBhQH0fO0R34Z0=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/shirou/gopsutil/v4 v4.25.1 h1:QSWkTc+fu9LTAWfkZwZ6j8MSUk4A2LV7rbH0ZqmLjXs=
github.com/shirou/gopsutil/v4 v4.25.1/go.mod h1:RoUCUpndaJFtT+2zsZzzmhvbfGoDCJ7nFXKJf8GqJbI=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
@@ -303,7 +365,13 @@ github.com/suzuki-shunsuke/pinact v1.6.0 h1:2QvSzREOquwLwKXhF9Hj0AInE/Rl63SZz9dK
github.com/suzuki-shunsuke/pinact v1.6.0/go.mod h1:FDUMck0mmL0mcnNZ23Vjh/aOR5cIdZhF1IIpGksT4dQ=
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4 h1:YGHgrVjGTYHY98II6zijXUHP+OyvrzSCvd8m9iUcaK8=
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4/go.mod h1:sSi6xaUaHfaqu32ECLeyE7NTMv+ZM5dW0JikhllaalY=
github.com/testcontainers/testcontainers-go v0.37.0 h1:L2Qc0vkTw2EHWQ08djon0D2uw7Z/PtHS/QzZZ5Ra/hg=
github.com/testcontainers/testcontainers-go v0.37.0/go.mod h1:QPzbxZhQ6Bclip9igjLFj6z0hs01bU8lrl2dHQmgFGM=
github.com/thlib/go-timezone-local v0.0.0-20210907160436-ef149e42d28e/go.mod h1:/Tnicc6m/lsJE0irFMA0LfIwTBo4QP7A8IfyIv4zZKI=
github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=
github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
github.com/ulikunitz/xz v0.5.12 h1:37Nm15o69RwBkXM0J6A5OlE67RZTfzUxTj8fB3dfcsc=
github.com/ulikunitz/xz v0.5.12/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/urfave/cli/v2 v2.27.6 h1:VdRdS98FNhKZ8/Az8B7MTyGQmpIr36O1EHybx/LaZ4g=
@@ -314,11 +382,19 @@ github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofm
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
gitlab.com/digitalxero/go-conventional-commit v1.0.7 h1:8/dO6WWG+98PMhlZowt/YjuiKhqhGlOCwlIV8SqqGh8=
gitlab.com/digitalxero/go-conventional-commit v1.0.7/go.mod h1:05Xc2BFsSyC5tKhK0y+P3bs0AwUtNuTp+mTpbCU/DZ0=
go.etcd.io/bbolt v1.4.2 h1:IrUHp260R8c+zYx/Tm8QZr04CX+qWS5PGfPdevhdm1I=
go.etcd.io/bbolt v1.4.2/go.mod h1:Is8rSHO/b4f3XigBC0lL0+4FwAQv3HXEEIgFMuKHceM=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 h1:jq9TW8u3so/bN+JPT166wjOI6/vQPF6Xe7nMNIltagk=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0/go.mod h1:p8pYQP+m5XfbZm9fxtSKAbM6oIllS7s2AfxrChvc7iw=
go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
go.opentelemetry.io/otel/metric v1.35.0 h1:0znxYu2SNyuMSQT4Y9WDWej0VpcsxkuklLa4/siN90M=
@@ -334,6 +410,7 @@ go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
go.yaml.in/yaml/v3 v3.0.3 h1:bXOww4E/J3f66rav3pX3m8w6jDE4knZjGOw8b5Y6iNE=
go.yaml.in/yaml/v3 v3.0.3/go.mod h1:tBHosrYAkRZjRAOREWbDnBXUf08JOwYq++0QNwQiWzI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
@@ -344,12 +421,16 @@ golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ=
golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.25.0 h1:n7a+ZbQKQA/Ysbyb0/6IbB1H/X41mKgbhfv7AfG/44w=
golang.org/x/mod v0.25.0/go.mod h1:IXM97Txy2VM4PJ3gI61r1YEk/gAj6zAHN3AdZt6S9Ww=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
@@ -360,20 +441,26 @@ golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
golang.org/x/oauth2 v0.28.0 h1:CrgCKl8PPAVtLnU3c+EDw6x11699EWlsDeWNWKdIOkc=
golang.org/x/oauth2 v0.28.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210319071255-635bc2c9138d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -384,6 +471,7 @@ golang.org/x/sys v0.0.0-20220818161305-2296e01440c6/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
@@ -408,6 +496,8 @@ golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo=
@@ -415,6 +505,7 @@ golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg
golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=

View File

@@ -1,6 +1,7 @@
package ogtags
import (
"context"
"errors"
"log/slog"
"net/url"
@@ -8,7 +9,7 @@ import (
)
// GetOGTags is the main function that retrieves Open Graph tags for a URL
func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]string, error) {
func (c *OGTagCache) GetOGTags(ctx context.Context, url *url.URL, originalHost string) (map[string]string, error) {
if url == nil {
return nil, errors.New("nil URL provided, cannot fetch OG tags")
}
@@ -21,12 +22,12 @@ func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]st
cacheKey := c.generateCacheKey(target, originalHost)
// Check cache first
if cachedTags := c.checkCache(cacheKey); cachedTags != nil {
if cachedTags := c.checkCache(ctx, cacheKey); cachedTags != nil {
return cachedTags, nil
}
// Fetch HTML content, passing the original host
doc, err := c.fetchHTMLDocumentWithCache(target, originalHost, cacheKey)
doc, err := c.fetchHTMLDocumentWithCache(ctx, target, originalHost, cacheKey)
if errors.Is(err, syscall.ECONNREFUSED) {
slog.Debug("Connection refused, returning empty tags")
return nil, nil
@@ -42,7 +43,7 @@ func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]st
ogTags := c.extractOGTags(doc)
// Store in cache
c.cache.Set(cacheKey, ogTags, c.ogTimeToLive)
c.cache.Set(ctx, cacheKey, ogTags, c.ogTimeToLive)
return ogTags, nil
}
@@ -59,8 +60,8 @@ func (c *OGTagCache) generateCacheKey(target string, originalHost string) string
}
// checkCache checks if we have the tags cached and returns them if so
func (c *OGTagCache) checkCache(cacheKey string) map[string]string {
if cachedTags, ok := c.cache.Get(cacheKey); ok {
func (c *OGTagCache) checkCache(ctx context.Context, cacheKey string) map[string]string {
if cachedTags, err := c.cache.Get(ctx, cacheKey); err == nil {
slog.Debug("cache hit", "tags", cachedTags)
return cachedTags
}

View File

@@ -9,6 +9,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestCacheReturnsDefault(t *testing.T) {
@@ -21,14 +22,14 @@ func TestCacheReturnsDefault(t *testing.T) {
TimeToLive: time.Minute,
ConsiderHost: false,
Override: want,
})
}, memory.New(t.Context()))
u, err := url.Parse("https://anubis.techaro.lol")
if err != nil {
t.Fatal(err)
}
result, err := cache.GetOGTags(u, "anubis.techaro.lol")
result, err := cache.GetOGTags(t.Context(), u, "anubis.techaro.lol")
if err != nil {
t.Fatal(err)
}
@@ -49,7 +50,7 @@ func TestCheckCache(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Set up test data
urlStr := "http://example.com/page"
@@ -60,16 +61,16 @@ func TestCheckCache(t *testing.T) {
cacheKey := cache.generateCacheKey(urlStr, "example.com")
// Test cache miss
tags := cache.checkCache(cacheKey)
tags := cache.checkCache(t.Context(), cacheKey)
if tags != nil {
t.Errorf("expected nil tags on cache miss, got %v", tags)
}
// Manually add to cache
cache.cache.Set(cacheKey, expectedTags, time.Minute)
cache.cache.Set(t.Context(), cacheKey, expectedTags, time.Minute)
// Test cache hit
tags = cache.checkCache(cacheKey)
tags = cache.checkCache(t.Context(), cacheKey)
if tags == nil {
t.Fatal("expected non-nil tags on cache hit, got nil")
}
@@ -112,7 +113,7 @@ func TestGetOGTags(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Parse the test server URL
parsedURL, err := url.Parse(ts.URL)
@@ -122,7 +123,7 @@ func TestGetOGTags(t *testing.T) {
// Test fetching OG tags from the test server
// Pass the host from the parsed test server URL
ogTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
ogTags, err := cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags: %v", err)
}
@@ -142,14 +143,14 @@ func TestGetOGTags(t *testing.T) {
// Test fetching OG tags from the cache
// Pass the host from the parsed test server URL
ogTags, err = cache.GetOGTags(parsedURL, parsedURL.Host)
ogTags, err = cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}
// Test fetching OG tags from the cache (3rd time)
// Pass the host from the parsed test server URL
newOgTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
newOgTags, err := cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}
@@ -263,10 +264,10 @@ func TestGetOGTagsWithHostConsideration(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: tc.ogCacheConsiderHost,
})
}, memory.New(t.Context()))
for i, req := range tc.requests {
ogTags, err := cache.GetOGTags(parsedURL, req.host)
ogTags, err := cache.GetOGTags(t.Context(), parsedURL, req.host)
if err != nil {
t.Errorf("Request %d (host: %s): unexpected error: %v", i+1, req.host, err)
continue // Skip further checks for this request if error occurred

View File

@@ -20,8 +20,8 @@ var (
// fetchHTMLDocumentWithCache fetches the HTML document from the given URL string,
// preserving the original host header.
func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost string, cacheKey string) (*html.Node, error) {
req, err := http.NewRequestWithContext(context.Background(), "GET", urlStr, nil)
func (c *OGTagCache) fetchHTMLDocumentWithCache(ctx context.Context, urlStr string, originalHost string, cacheKey string) (*html.Node, error) {
req, err := http.NewRequestWithContext(ctx, "GET", urlStr, nil)
if err != nil {
return nil, fmt.Errorf("failed to create http request: %w", err)
}
@@ -41,7 +41,7 @@ func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost stri
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
slog.Debug("og: request timed out", "url", urlStr)
c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
c.cache.Set(ctx, cacheKey, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
}
return nil, fmt.Errorf("http get failed: %w", err)
}
@@ -56,7 +56,7 @@ func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost stri
if resp.StatusCode != http.StatusOK {
slog.Debug("og: received non-OK status code", "url", urlStr, "status", resp.StatusCode)
c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
c.cache.Set(ctx, cacheKey, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
return nil, fmt.Errorf("%w: page not found", ErrOgHandled)
}

View File

@@ -1,6 +1,7 @@
package ogtags
import (
"context"
"fmt"
"io"
"net/http"
@@ -11,6 +12,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -85,8 +87,8 @@ func TestFetchHTMLDocument(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
doc, err := cache.fetchHTMLDocument(ts.URL, "anything")
}, memory.New(t.Context()))
doc, err := cache.fetchHTMLDocument(t.Context(), ts.URL, "anything")
if tt.expectError {
if err == nil {
@@ -116,9 +118,9 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
doc, err := cache.fetchHTMLDocument("http://invalid.url.that.doesnt.exist.example", "anything")
doc, err := cache.fetchHTMLDocument(t.Context(), "http://invalid.url.that.doesnt.exist.example", "anything")
if err == nil {
t.Error("expected error for invalid URL, got nil")
@@ -130,7 +132,7 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
}
// fetchHTMLDocument allows you to call fetchHTMLDocumentWithCache without a duplicate generateCacheKey call
func (c *OGTagCache) fetchHTMLDocument(urlStr string, originalHost string) (*html.Node, error) {
func (c *OGTagCache) fetchHTMLDocument(ctx context.Context, urlStr string, originalHost string) (*html.Node, error) {
cacheKey := c.generateCacheKey(urlStr, originalHost)
return c.fetchHTMLDocumentWithCache(urlStr, originalHost, cacheKey)
return c.fetchHTMLDocumentWithCache(ctx, urlStr, originalHost, cacheKey)
}

View File

@@ -8,6 +8,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestIntegrationGetOGTags(t *testing.T) {
@@ -110,7 +111,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Create URL for test
testURL, _ := url.Parse(ts.URL)
@@ -119,7 +120,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
// Get OG tags
// Pass the host from the test URL
ogTags, err := cache.GetOGTags(testURL, testURL.Host)
ogTags, err := cache.GetOGTags(t.Context(), testURL, testURL.Host)
// Check error expectation
if tc.expectError {
@@ -147,7 +148,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
// Test cache retrieval
// Pass the host from the test URL
cachedOGTags, err := cache.GetOGTags(testURL, testURL.Host)
cachedOGTags, err := cache.GetOGTags(t.Context(), testURL, testURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}

View File

@@ -7,6 +7,7 @@ import (
"testing"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -30,7 +31,7 @@ func BenchmarkGetTarget(b *testing.B) {
for _, tt := range tests {
b.Run(tt.name, func(b *testing.B) {
cache := NewOGTagCache(tt.target, config.OpenGraph{})
cache := NewOGTagCache(tt.target, config.OpenGraph{}, memory.New(b.Context()))
urls := make([]*url.URL, len(tt.paths))
for i, path := range tt.paths {
u, _ := url.Parse(path)
@@ -66,7 +67,7 @@ func BenchmarkExtractOGTags(b *testing.B) {
</head><body><div><p>Content</p></div></body></html>`,
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(b.Context()))
docs := make([]*html.Node, len(htmlSamples))
for i, sample := range htmlSamples {
@@ -84,7 +85,7 @@ func BenchmarkExtractOGTags(b *testing.B) {
// Memory usage test
func TestMemoryUsage(t *testing.T) {
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(t.Context()))
// Force GC and wait for it to complete
runtime.GC()

View File

@@ -9,8 +9,8 @@ import (
"strings"
"time"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
)
const (
@@ -22,7 +22,7 @@ const (
)
type OGTagCache struct {
cache *decaymap.Impl[string, map[string]string]
cache store.JSON[map[string]string]
targetURL *url.URL
client *http.Client
@@ -36,7 +36,7 @@ type OGTagCache struct {
ogOverride map[string]string
}
func NewOGTagCache(target string, conf config.OpenGraph) *OGTagCache {
func NewOGTagCache(target string, conf config.OpenGraph, backend store.Interface) *OGTagCache {
// Predefined approved tags and prefixes
defaultApprovedTags := []string{"description", "keywords", "author"}
defaultApprovedPrefixes := []string{"og:", "twitter:", "fediverse:"}
@@ -77,7 +77,10 @@ func NewOGTagCache(target string, conf config.OpenGraph) *OGTagCache {
}
return &OGTagCache{
cache: decaymap.New[string, map[string]string](),
cache: store.JSON[map[string]string]{
Underlying: backend,
Prefix: "ogtags:",
},
targetURL: parsedTargetURL,
ogPassthrough: conf.Enabled,
ogTimeToLive: conf.TimeToLive,
@@ -124,9 +127,3 @@ func (c *OGTagCache) getTarget(u *url.URL) string {
return sb.String()
}
func (c *OGTagCache) Cleanup() {
if c.cache != nil {
c.cache.Cleanup()
}
}

View File

@@ -1,12 +1,14 @@
package ogtags
import (
"context"
"net/url"
"strings"
"testing"
"unicode/utf8"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -46,7 +48,7 @@ func FuzzGetTarget(f *testing.F) {
}
// Create cache - should not panic
cache := NewOGTagCache(target, config.OpenGraph{})
cache := NewOGTagCache(target, config.OpenGraph{}, memory.New(context.Background()))
// Create URL
u := &url.URL{
@@ -130,7 +132,7 @@ func FuzzExtractOGTags(f *testing.F) {
return
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(context.Background()))
// Should not panic
tags := cache.extractOGTags(doc)
@@ -186,7 +188,7 @@ func FuzzGetTargetRoundTrip(f *testing.F) {
t.Skip()
}
cache := NewOGTagCache(target, config.OpenGraph{})
cache := NewOGTagCache(target, config.OpenGraph{}, memory.New(context.Background()))
u := &url.URL{Path: path, RawQuery: query}
result := cache.getTarget(u)
@@ -243,7 +245,7 @@ func FuzzExtractMetaTagInfo(f *testing.F) {
},
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(context.Background()))
// Should not panic
property, content := cache.extractMetaTagInfo(node)
@@ -296,7 +298,7 @@ func BenchmarkFuzzedGetTarget(b *testing.B) {
for _, input := range inputs {
b.Run(input.name, func(b *testing.B) {
cache := NewOGTagCache(input.target, config.OpenGraph{})
cache := NewOGTagCache(input.target, config.OpenGraph{}, memory.New(context.Background()))
u := &url.URL{Path: input.path, RawQuery: input.query}
b.ResetTimer()

View File

@@ -15,6 +15,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestNewOGTagCache(t *testing.T) {
@@ -44,7 +45,7 @@ func TestNewOGTagCache(t *testing.T) {
Enabled: tt.ogPassthrough,
TimeToLive: tt.ogTimeToLive,
ConsiderHost: false,
})
}, memory.New(t.Context()))
if cache == nil {
t.Fatal("expected non-nil cache, got nil")
@@ -84,7 +85,7 @@ func TestNewOGTagCache_UnixSocket(t *testing.T) {
Enabled: true,
TimeToLive: 5 * time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
if cache == nil {
t.Fatal("expected non-nil cache, got nil")
@@ -169,7 +170,7 @@ func TestGetTarget(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
u := &url.URL{
Path: tt.path,
@@ -242,14 +243,14 @@ func TestIntegrationGetOGTags_UnixSocket(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Create a dummy URL for the request (path and query matter)
testReqURL, _ := url.Parse("/some/page?query=1")
// Get OG tags
// Pass an empty string for host, as it's irrelevant for unix sockets
ogTags, err := cache.GetOGTags(testReqURL, "")
ogTags, err := cache.GetOGTags(t.Context(), testReqURL, "")
if err != nil {
t.Fatalf("GetOGTags failed for unix socket: %v", err)
@@ -265,7 +266,7 @@ func TestIntegrationGetOGTags_UnixSocket(t *testing.T) {
// Test cache retrieval (should hit cache)
// Pass an empty string for host
cachedTags, err := cache.GetOGTags(testReqURL, "")
cachedTags, err := cache.GetOGTags(t.Context(), testReqURL, "")
if err != nil {
t.Fatalf("GetOGTags (cache hit) failed for unix socket: %v", err)
}

View File

@@ -7,6 +7,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -17,7 +18,7 @@ func TestExtractOGTags(t *testing.T) {
Enabled: false,
ConsiderHost: false,
TimeToLive: time.Minute,
})
}, memory.New(t.Context()))
// Manually set approved tags/prefixes based on the user request for clarity
testCache.approvedTags = []string{"description"}
testCache.approvedPrefixes = []string{"og:"}
@@ -198,7 +199,7 @@ func TestExtractMetaTagInfo(t *testing.T) {
Enabled: false,
ConsiderHost: false,
TimeToLive: time.Minute,
})
}, memory.New(t.Context()))
testCache.approvedTags = []string{"description"}
testCache.approvedPrefixes = []string{"og:"}

22
internal/unbreakdocker.go Normal file
View File

@@ -0,0 +1,22 @@
package internal
import (
"os"
"os/exec"
)
func UnbreakDocker() {
// XXX(Xe): This is bad code. Do not do this.
//
// I have to do this because I'm running from inside the context of a dev
// container. This dev container runs in a different docker network than
// the valkey test container runs in. In order to let my dev container
// connect to the test container, they need to share a network in common.
// The easiest network to use for this is the default "bridge" network.
//
// This is a horrifying monstrosity, but the part that scares me the most
// is the fact that it works.
if hostname, err := os.Hostname(); err == nil {
exec.Command("docker", "network", "connect", "bridge", hostname).Run()
}
}

View File

@@ -1,8 +1,9 @@
package lib
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"crypto/rand"
"encoding/json"
"errors"
"fmt"
@@ -16,6 +17,7 @@ import (
"github.com/golang-jwt/jwt/v5"
"github.com/google/cel-go/common/types"
"github.com/google/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
@@ -30,6 +32,7 @@ import (
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/checker"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
// challenge implementations
_ "github.com/TecharoHQ/anubis/lib/challenge/metarefresh"
@@ -67,11 +70,11 @@ type Server struct {
next http.Handler
mux *http.ServeMux
policy *policy.ParsedConfig
DNSBLCache *decaymap.Impl[string, dnsbl.DroneBLResponse]
OGTags *ogtags.OGTagCache
ed25519Priv ed25519.PrivateKey
hs512Secret []byte
opts Options
store store.Interface
}
func (s *Server) getTokenKeyfunc() jwt.Keyfunc {
@@ -87,23 +90,51 @@ func (s *Server) getTokenKeyfunc() jwt.Keyfunc {
}
}
func (s *Server) challengeFor(r *http.Request, difficulty int) string {
var fp [32]byte
if len(s.hs512Secret) == 0 {
fp = sha256.Sum256(s.ed25519Priv.Public().(ed25519.PublicKey)[:])
} else {
fp = sha256.Sum256(s.hs512Secret)
func (s *Server) challengeFor(r *http.Request) (*challenge.Challenge, error) {
ckies := r.CookiesNamed(anubis.TestCookieName)
if len(ckies) == 0 {
return s.issueChallenge(r.Context(), r)
}
challengeData := fmt.Sprintf(
"X-Real-IP=%s,User-Agent=%s,WeekTime=%s,Fingerprint=%x,Difficulty=%d",
r.Header.Get("X-Real-Ip"),
r.UserAgent(),
time.Now().UTC().Round(24*7*time.Hour).Format(time.RFC3339),
fp,
difficulty,
)
return internal.FastHash(challengeData)
j := store.JSON[challenge.Challenge]{Underlying: s.store}
ckie := ckies[0]
chall, err := j.Get(r.Context(), "challenge:"+ckie.Value)
if err != nil {
return nil, err
}
return &chall, nil
}
func (s *Server) issueChallenge(ctx context.Context, r *http.Request) (*challenge.Challenge, error) {
id, err := uuid.NewV7()
if err != nil {
return nil, err
}
var randomData = make([]byte, 256)
if _, err := rand.Read(randomData); err != nil {
return nil, err
}
chall := challenge.Challenge{
ID: id.String(),
RandomData: fmt.Sprintf("%x", randomData),
IssuedAt: time.Now(),
Metadata: map[string]string{
"User-Agent": r.Header.Get("User-Agent"),
"X-Real-Ip": r.Header.Get("X-Real-Ip"),
},
}
j := store.JSON[challenge.Challenge]{Underlying: s.store}
if err := j.Set(ctx, "challenge:"+id.String(), chall, 30*time.Minute); err != nil {
return nil, err
}
return &chall, err
}
func (s *Server) maybeReverseProxyHttpStatusOnly(w http.ResponseWriter, r *http.Request) {
@@ -247,15 +278,16 @@ func (s *Server) checkRules(w http.ResponseWriter, r *http.Request, cr policy.Ch
}
func (s *Server) handleDNSBL(w http.ResponseWriter, r *http.Request, ip string, lg *slog.Logger) bool {
db := &store.JSON[dnsbl.DroneBLResponse]{Underlying: s.store, Prefix: "dronebl:"}
if s.policy.DNSBL && ip != "" {
resp, ok := s.DNSBLCache.Get(ip)
if !ok {
resp, err := db.Get(r.Context(), ip)
if err != nil {
lg.Debug("looking up ip in dnsbl")
resp, err := dnsbl.Lookup(ip)
if err != nil {
lg.Error("can't look up ip in dnsbl", "err", err)
}
s.DNSBLCache.Set(ip, resp, 24*time.Hour)
db.Set(r.Context(), ip, resp, 24*time.Hour)
droneBLHits.WithLabelValues(resp.String()).Inc()
}
@@ -309,15 +341,30 @@ func (s *Server) MakeChallenge(w http.ResponseWriter, r *http.Request) {
return
}
lg = lg.With("check_result", cr)
chal := s.challengeFor(r, rule.Challenge.Difficulty)
s.SetCookie(w, CookieOpts{Host: r.Host, Name: anubis.TestCookieName, Value: chal})
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("failed to fetch or issue challenge", "err", err)
w.WriteHeader(http.StatusInternalServerError)
err := encoder.Encode(struct {
Error string `json:"error"`
}{
Error: fmt.Sprintf("%s \"makeChallenge\"", localizer.T("internal_server_error")),
})
if err != nil {
lg.Error("failed to encode error response", "err", err)
w.WriteHeader(http.StatusInternalServerError)
}
return
}
s.SetCookie(w, CookieOpts{Host: r.Host, Name: anubis.TestCookieName, Value: chall.ID})
err = encoder.Encode(struct {
Rules *config.ChallengeRules `json:"rules"`
Challenge string `json:"challenge"`
}{
Challenge: chal,
Challenge: chall.RandomData,
Rules: rule.Challenge,
})
if err != nil {
@@ -325,7 +372,7 @@ func (s *Server) MakeChallenge(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
return
}
lg.Debug("made challenge", "challenge", chal, "rules", rule.Challenge, "cr", cr)
lg.Debug("made challenge", "challenge", chall, "rules", rule.Challenge, "cr", cr)
challengesIssued.WithLabelValues("api").Inc()
}
@@ -384,9 +431,20 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
return
}
challengeStr := s.challengeFor(r, rule.Challenge.Difficulty)
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("check failed", "err", err)
s.respondWithError(w, r, fmt.Sprintf("%s: %s", localizer.T("internal_server_error"), rule.Challenge.Algorithm))
return
}
if err := impl.Validate(r, lg, rule, challengeStr); err != nil {
in := &challenge.ValidateInput{
Challenge: chall,
Rule: rule,
Store: s.store,
}
if err := impl.Validate(r, lg, in); err != nil {
failedValidations.WithLabelValues(rule.Challenge.Algorithm).Inc()
var cerr *challenge.Error
s.ClearCookie(w, CookieOpts{Path: cookiePath, Host: r.Host})
@@ -405,7 +463,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
// generate JWT cookie
tokenString, err := s.signJWT(jwt.MapClaims{
"challenge": challengeStr,
"challenge": chall.ID,
"method": rule.Challenge.Algorithm,
"policyRule": rule.Hash(),
"action": string(cr.Rule),
@@ -493,8 +551,3 @@ func (s *Server) check(r *http.Request) (policy.CheckResult, *policy.Bot, error)
Rules: &checker.List{},
}, nil
}
func (s *Server) CleanupDecayMap() {
s.DNSBLCache.Cleanup()
s.OGTags.Cleanup()
}

View File

@@ -1,60 +1,11 @@
package challenge
import (
"log/slog"
"net/http"
"sort"
"sync"
import "time"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/a-h/templ"
)
var (
registry map[string]Impl = map[string]Impl{}
regLock sync.RWMutex
)
func Register(name string, impl Impl) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Impl, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}
type IssueInput struct {
Impressum *config.Impressum
Rule *policy.Bot
Challenge string
OGTags map[string]string
}
type Impl interface {
// Setup registers any additional routes with the Impl for assets or API routes.
Setup(mux *http.ServeMux)
// Issue a new challenge to the user, called by the Anubis.
Issue(r *http.Request, lg *slog.Logger, in *IssueInput) (templ.Component, error)
// Validate a challenge, making sure that it passes muster.
Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, challenge string) error
// Challenge is the metadata about a single challenge issuance.
type Challenge struct {
ID string `json:"id"` // UUID identifying the challenge
RandomData string `json:"randomData"` // The random data the client processes
IssuedAt time.Time `json:"issuedAt"` // When the challenge was issued
Metadata map[string]string `json:"metadata"` // Challenge metadata such as IP address and user agent
}

View File

@@ -0,0 +1,23 @@
package challengetest
import (
"testing"
"time"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/google/uuid"
)
func New(t *testing.T) *challenge.Challenge {
t.Helper()
id := uuid.Must(uuid.NewV7())
randomData := internal.SHA256sum(time.Now().String())
return &challenge.Challenge{
ID: id.String(),
RandomData: randomData,
IssuedAt: time.Now(),
}
}

View File

@@ -0,0 +1,7 @@
package challengetest
import "testing"
func TestNew(t *testing.T) {
_ = New(t)
}

View File

@@ -0,0 +1,68 @@
package challenge
import (
"log/slog"
"net/http"
"sort"
"sync"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/a-h/templ"
)
var (
registry map[string]Impl = map[string]Impl{}
regLock sync.RWMutex
)
func Register(name string, impl Impl) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Impl, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}
type IssueInput struct {
Impressum *config.Impressum
Rule *policy.Bot
Challenge *Challenge
OGTags map[string]string
Store store.Interface
}
type ValidateInput struct {
Rule *policy.Bot
Challenge *Challenge
Store store.Interface
}
type Impl interface {
// Setup registers any additional routes with the Impl for assets or API routes.
Setup(mux *http.ServeMux)
// Issue a new challenge to the user, called by the Anubis.
Issue(r *http.Request, lg *slog.Logger, in *IssueInput) (templ.Component, error)
// Validate a challenge, making sure that it passes muster.
Validate(r *http.Request, lg *slog.Logger, in *ValidateInput) error
}

View File

@@ -9,7 +9,6 @@ import (
"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/web"
"github.com/a-h/templ"
)
@@ -32,11 +31,11 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *challenge.IssueInput)
q := u.Query()
q.Set("redir", r.URL.String())
q.Set("challenge", in.Challenge)
q.Set("challenge", in.Challenge.RandomData)
u.RawQuery = q.Encode()
loc := localization.GetLocalizer(r)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), page(in.Challenge, u.String(), in.Rule.Challenge.Difficulty, loc), in.Impressum, in.Challenge, in.Rule.Challenge, in.OGTags, loc)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), page(u.String(), in.Rule.Challenge.Difficulty, loc), in.Impressum, in.Challenge.RandomData, in.Rule.Challenge, in.OGTags, loc)
if err != nil {
return nil, fmt.Errorf("can't render page: %w", err)
@@ -45,11 +44,11 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *challenge.IssueInput)
return component, nil
}
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, wantChallenge string) error {
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, in *challenge.ValidateInput) error {
gotChallenge := r.FormValue("challenge")
if subtle.ConstantTimeCompare([]byte(wantChallenge), []byte(gotChallenge)) != 1 {
return challenge.NewError("validate", "invalid response", fmt.Errorf("%w: wanted response %s but got %s", challenge.ErrFailed, wantChallenge, gotChallenge))
if subtle.ConstantTimeCompare([]byte(in.Challenge.RandomData), []byte(gotChallenge)) != 1 {
return challenge.NewError("validate", "invalid response", fmt.Errorf("%w: wanted response %s but got %s", challenge.ErrFailed, in.Challenge.RandomData, gotChallenge))
}
return nil

View File

@@ -7,7 +7,7 @@ import (
"github.com/TecharoHQ/anubis/lib/localization"
)
templ page(challenge, redir string, difficulty int, loc *localization.SimpleLocalizer) {
templ page(redir string, difficulty int, loc *localization.SimpleLocalizer) {
<div class="centered-div">
<img id="image" style="width:100%;max-width:256px;" src={ anubis.BasePrefix + "/.within.website/x/cmd/anubis/static/img/pensive.webp?cacheBuster=" + anubis.Version }/>
<img style="display:none;" style="width:100%;max-width:256px;" src={ anubis.BasePrefix + "/.within.website/x/cmd/anubis/static/img/happy.webp?cacheBuster=" + anubis.Version }/>

View File

@@ -15,7 +15,7 @@ import (
"github.com/TecharoHQ/anubis/lib/localization"
)
func page(challenge, redir string, difficulty int, loc *localization.SimpleLocalizer) templ.Component {
func page(redir string, difficulty int, loc *localization.SimpleLocalizer) templ.Component {
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {

View File

@@ -11,7 +11,6 @@ import (
"github.com/TecharoHQ/anubis/internal"
chall "github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/web"
"github.com/a-h/templ"
)
@@ -31,7 +30,7 @@ func (i *Impl) Setup(mux *http.ServeMux) {
func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *chall.IssueInput) (templ.Component, error) {
loc := localization.GetLocalizer(r)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), web.Index(loc), in.Impressum, in.Challenge, in.Rule.Challenge, in.OGTags, loc)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), web.Index(loc), in.Impressum, in.Challenge.RandomData, in.Rule.Challenge, in.OGTags, loc)
if err != nil {
return nil, fmt.Errorf("can't render page: %w", err)
}
@@ -39,7 +38,10 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *chall.IssueInput) (te
return component, nil
}
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, challenge string) error {
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, in *chall.ValidateInput) error {
rule := in.Rule
challenge := in.Challenge.RandomData
nonceStr := r.FormValue("nonce")
if nonceStr == "" {
return chall.NewError("validate", "invalid response", fmt.Errorf("%w nonce", chall.ErrMissingField))

View File

@@ -124,16 +124,25 @@ func TestBasic(t *testing.T) {
t.Run(cs.name, func(t *testing.T) {
lg := slog.With()
i.Setup(http.NewServeMux())
inp := &challenge.IssueInput{
Rule: bot,
Challenge: cs.challengeStr,
Rule: bot,
Challenge: &challenge.Challenge{
RandomData: cs.challengeStr,
},
}
if _, err := i.Issue(cs.req, lg, inp); err != nil {
t.Errorf("can't issue challenge: %v", err)
}
if err := i.Validate(cs.req, lg, bot, cs.challengeStr); !errors.Is(err, cs.err) {
if err := i.Validate(cs.req, lg, &challenge.ValidateInput{
Rule: bot,
Challenge: &challenge.Challenge{
RandomData: cs.challengeStr,
},
}); !errors.Is(err, cs.err) {
t.Errorf("got wrong error from Validate, got %v but wanted %v", err, cs.err)
}
})

View File

@@ -15,9 +15,7 @@ import (
"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/data"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/internal/dnsbl"
"github.com/TecharoHQ/anubis/internal/ogtags"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
@@ -108,8 +106,8 @@ func New(opts Options) (*Server, error) {
hs512Secret: opts.HS512Secret,
policy: opts.Policy,
opts: opts,
DNSBLCache: decaymap.New[string, dnsbl.DroneBLResponse](),
OGTags: ogtags.NewOGTagCache(opts.Target, opts.Policy.OpenGraph),
OGTags: ogtags.NewOGTagCache(opts.Target, opts.Policy.OpenGraph, opts.Policy.Store),
store: opts.Policy.Store,
}
mux := http.NewServeMux()

View File

@@ -128,19 +128,24 @@ func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, rule *polic
}
challengesIssued.WithLabelValues("embedded").Add(1)
challengeStr := s.challengeFor(r, rule.Challenge.Difficulty)
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("can't get challenge", "err", "err")
s.respondWithError(w, r, fmt.Sprintf("%s: %s", localizer.T("internal_server_error"), rule.Challenge.Algorithm))
return
}
var ogTags map[string]string = nil
if s.opts.OpenGraph.Enabled {
var err error
ogTags, err = s.OGTags.GetOGTags(r.URL, r.Host)
ogTags, err = s.OGTags.GetOGTags(r.Context(), r.URL, r.Host)
if err != nil {
lg.Error("failed to get OG tags", "err", err)
}
}
s.SetCookie(w, CookieOpts{
Value: challengeStr,
Value: chall.ID,
Host: r.Host,
Path: "/",
Name: anubis.TestCookieName,
@@ -157,8 +162,9 @@ func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, rule *polic
in := &challenge.IssueInput{
Impressum: s.policy.Impressum,
Rule: rule,
Challenge: challengeStr,
Challenge: chall,
OGTags: ogTags,
Store: s.store,
}
component, err := impl.Issue(r, lg, in)

View File

@@ -329,6 +329,7 @@ type fileConfig struct {
OpenGraph openGraphFileConfig `json:"openGraph,omitempty"`
Impressum *Impressum `json:"impressum,omitempty"`
StatusCodes StatusCodes `json:"status_codes"`
Store *Store `json:"store"`
Thresholds []Threshold `json:"thresholds"`
}
@@ -361,6 +362,12 @@ func (c *fileConfig) Valid() error {
}
}
if c.Store != nil {
if err := c.Store.Valid(); err != nil {
errs = append(errs, err)
}
}
if len(errs) != 0 {
return fmt.Errorf("config is not valid:\n%w", errors.Join(errs...))
}
@@ -374,6 +381,9 @@ func Load(fin io.Reader, fname string) (*Config, error) {
Challenge: http.StatusOK,
Deny: http.StatusOK,
},
Store: &Store{
Backend: "memory",
},
}
if err := yaml.NewYAMLToJSONDecoder(fin).Decode(&c); err != nil {
@@ -392,6 +402,7 @@ func Load(fin io.Reader, fname string) (*Config, error) {
Override: c.OpenGraph.Override,
},
StatusCodes: c.StatusCodes,
Store: c.Store,
}
if c.OpenGraph.TimeToLive != "" {
@@ -457,6 +468,7 @@ type Config struct {
Impressum *Impressum
OpenGraph OpenGraph
StatusCodes StatusCodes
Store *Store
}
func (c Config) Valid() error {

View File

@@ -1,4 +1,4 @@
package config
package config_test
import (
"errors"
@@ -8,6 +8,7 @@ import (
"testing"
"github.com/TecharoHQ/anubis/data"
. "github.com/TecharoHQ/anubis/lib/policy/config"
)
func p[V any](v V) *V { return &v }
@@ -325,37 +326,37 @@ func TestConfigValidBad(t *testing.T) {
func TestBotConfigZero(t *testing.T) {
var b BotConfig
if !b.Zero() {
t.Error("zero value BotConfig is not zero value")
t.Error("zero value config.BotConfig is not zero value")
}
b.Name = "hi"
if b.Zero() {
t.Error("BotConfig with name is zero value")
t.Error("config.BotConfig with name is zero value")
}
b.UserAgentRegex = p(".*")
if b.Zero() {
t.Error("BotConfig with user agent regex is zero value")
t.Error("config.BotConfig with user agent regex is zero value")
}
b.PathRegex = p(".*")
if b.Zero() {
t.Error("BotConfig with path regex is zero value")
t.Error("config.BotConfig with path regex is zero value")
}
b.HeadersRegex = map[string]string{"hi": "there"}
if b.Zero() {
t.Error("BotConfig with headers regex is zero value")
t.Error("config.BotConfig with headers regex is zero value")
}
b.Action = RuleAllow
if b.Zero() {
t.Error("BotConfig with action is zero value")
t.Error("config.BotConfig with action is zero value")
}
b.RemoteAddr = []string{"::/0"}
if b.Zero() {
t.Error("BotConfig with remote addresses is zero value")
t.Error("config.BotConfig with remote addresses is zero value")
}
b.Challenge = &ChallengeRules{
@@ -364,6 +365,6 @@ func TestBotConfigZero(t *testing.T) {
Algorithm: DefaultAlgorithm,
}
if b.Zero() {
t.Error("BotConfig with challenge rules is zero value")
t.Error("config.BotConfig with challenge rules is zero value")
}
}

View File

@@ -0,0 +1,44 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"github.com/TecharoHQ/anubis/lib/store"
_ "github.com/TecharoHQ/anubis/lib/store/all"
)
var (
ErrNoStoreBackend = errors.New("config.Store: no backend defined")
ErrUnknownStoreBackend = errors.New("config.Store: unknown backend")
)
type Store struct {
Backend string `json:"backend"`
Parameters json.RawMessage `json:"parameters"`
}
func (s *Store) Valid() error {
var errs []error
if len(s.Backend) == 0 {
errs = append(errs, ErrNoStoreBackend)
}
fac, ok := store.Get(s.Backend)
switch ok {
case true:
if err := fac.Valid(s.Parameters); err != nil {
errs = append(errs, err)
}
case false:
errs = append(errs, fmt.Errorf("%w: %q", ErrUnknownStoreBackend, s.Backend))
}
if len(errs) != 0 {
return errors.Join(errs...)
}
return nil
}

View File

@@ -0,0 +1,84 @@
package config_test
import (
"encoding/json"
"errors"
"testing"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/bbolt"
"github.com/TecharoHQ/anubis/lib/store/valkey"
)
func TestStoreValid(t *testing.T) {
for _, tt := range []struct {
name string
input config.Store
err error
}{
{
name: "no backend",
input: config.Store{},
err: config.ErrNoStoreBackend,
},
{
name: "in-memory backend",
input: config.Store{
Backend: "memory",
},
},
{
name: "bbolt backend",
input: config.Store{
Backend: "bbolt",
Parameters: json.RawMessage(`{"path": "/tmp/foo", "bucket": "bar"}`),
},
},
{
name: "valkey backend",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{"url": "redis://valkey:6379/0"}`),
},
},
{
name: "valkey backend no URL",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{}`),
},
err: valkey.ErrNoURL,
},
{
name: "valkey backend bad URL",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{"url": "http://anubis.techaro.lol"}`),
},
err: valkey.ErrBadURL,
},
{
name: "bbolt backend no path",
input: config.Store{
Backend: "bbolt",
Parameters: json.RawMessage(`{"path": "", "bucket": "bar"}`),
},
err: bbolt.ErrMissingPath,
},
{
name: "unknown backend",
input: config.Store{
Backend: "taco salad",
},
err: config.ErrUnknownStoreBackend,
},
} {
t.Run(tt.name, func(t *testing.T) {
if err := tt.input.Valid(); !errors.Is(err, tt.err) {
t.Logf("want: %v", tt.err)
t.Logf("got: %v", err)
t.Error("invalid error returned")
}
})
}
}

View File

@@ -11,8 +11,11 @@ import (
"github.com/TecharoHQ/anubis/internal/thoth"
"github.com/TecharoHQ/anubis/lib/policy/checker"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
_ "github.com/TecharoHQ/anubis/lib/store/all"
)
var (
@@ -35,9 +38,10 @@ type ParsedConfig struct {
OpenGraph config.OpenGraph
DefaultDifficulty int
StatusCodes config.StatusCodes
Store store.Interface
}
func NewParsedConfig(orig *config.Config) *ParsedConfig {
func newParsedConfig(orig *config.Config) *ParsedConfig {
return &ParsedConfig{
orig: orig,
OpenGraph: orig.OpenGraph,
@@ -55,7 +59,7 @@ func ParseConfig(ctx context.Context, fin io.Reader, fname string, defaultDiffic
tc, hasThothClient := thoth.FromContext(ctx)
result := NewParsedConfig(c)
result := newParsedConfig(c)
result.DefaultDifficulty = defaultDifficulty
for _, b := range c.Bots {
@@ -178,6 +182,19 @@ func ParseConfig(ctx context.Context, fin io.Reader, fname string, defaultDiffic
result.Thresholds = append(result.Thresholds, threshold)
}
stFac, ok := store.Get(c.Store.Backend)
switch ok {
case true:
store, err := stFac.Build(ctx, c.Store.Parameters)
if err != nil {
validationErrs = append(validationErrs, err)
} else {
result.Store = store
}
case false:
validationErrs = append(validationErrs, config.ErrUnknownStoreBackend)
}
if len(validationErrs) > 0 {
return nil, fmt.Errorf("errors validating policy config JSON %s: %w", fname, errors.Join(validationErrs...))
}

10
lib/store/all/all.go Normal file
View File

@@ -0,0 +1,10 @@
// Package all is a meta-package that imports all store implementations.
//
// This is a HACK to make tests work consistently.
package all
import (
_ "github.com/TecharoHQ/anubis/lib/store/bbolt"
_ "github.com/TecharoHQ/anubis/lib/store/memory"
_ "github.com/TecharoHQ/anubis/lib/store/valkey"
)

142
lib/store/bbolt/bbolt.go Normal file
View File

@@ -0,0 +1,142 @@
package bbolt
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"time"
"github.com/TecharoHQ/anubis/lib/store"
"go.etcd.io/bbolt"
)
var (
ErrBucketDoesNotExist = errors.New("bbolt: bucket does not exist")
ErrNotExists = errors.New("bbolt: value does not exist in store")
)
type Item struct {
Data []byte `json:"data"`
Expires time.Time `json:"expires"`
}
type Store struct {
bucket []byte
bdb *bbolt.DB
}
func (s *Store) Delete(ctx context.Context, key string) error {
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
if bkt.Get([]byte(key)) == nil {
return fmt.Errorf("%w: %q", ErrNotExists, key)
}
return bkt.Delete([]byte(key))
})
}
func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
var i Item
if err := s.bdb.View(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
bucketData := bkt.Get([]byte(key))
if bucketData == nil {
return fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
if err := json.Unmarshal(bucketData, &i); err != nil {
return fmt.Errorf("%w: %w", store.ErrCantDecode, err)
}
return nil
}); err != nil {
return nil, err
}
if time.Now().After(i.Expires) {
go s.Delete(context.Background(), key)
return nil, fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return i.Data, nil
}
func (s *Store) Set(ctx context.Context, key string, value []byte, expiry time.Duration) error {
i := Item{
Data: value,
Expires: time.Now().Add(expiry),
}
data, err := json.Marshal(i)
if err != nil {
return fmt.Errorf("%w: %w", store.ErrCantEncode, err)
}
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
return bkt.Put([]byte(key), data)
})
}
func (s *Store) cleanup(ctx context.Context) error {
now := time.Now()
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("cache bucket %q does not exist", string(s.bucket))
}
return bkt.ForEach(func(k, v []byte) error {
var i Item
data := bkt.Get(k)
if data == nil {
return fmt.Errorf("%s in Cache bucket does not exist???", string(k))
}
if err := json.Unmarshal(data, &i); err != nil {
return fmt.Errorf("can't unmarshal data at key %s: %w", string(k), err)
}
if now.After(i.Expires) {
return bkt.Delete(k)
}
return nil
})
})
}
func (s *Store) cleanupThread(ctx context.Context) {
t := time.NewTicker(5 * time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if err := s.cleanup(ctx); err != nil {
slog.Error("error during bbolt cleanup", "err", err)
}
}
}
}

View File

@@ -0,0 +1,23 @@
package bbolt
import (
"encoding/json"
"path/filepath"
"testing"
"github.com/TecharoHQ/anubis/lib/store/storetest"
)
func TestImpl(t *testing.T) {
path := filepath.Join(t.TempDir(), "db")
t.Log(path)
data, err := json.Marshal(Config{
Path: path,
Bucket: "anubis",
})
if err != nil {
t.Fatal(err)
}
storetest.Common(t, Factory{}, json.RawMessage(data))
}

100
lib/store/bbolt/factory.go Normal file
View File

@@ -0,0 +1,100 @@
package bbolt
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"github.com/TecharoHQ/anubis/lib/store"
"go.etcd.io/bbolt"
)
var (
ErrMissingPath = errors.New("bbolt: path is missing from config")
ErrCantWriteToPath = errors.New("bbolt: can't write to path")
)
func init() {
store.Register("bbolt", Factory{})
}
type Factory struct{}
func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface, error) {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if config.Bucket == "" {
config.Bucket = "anubis"
}
bdb, err := bbolt.Open(config.Path, 0600, nil)
if err != nil {
return nil, fmt.Errorf("can't open bbolt database %s: %w", config.Path, err)
}
if err := bdb.Update(func(tx *bbolt.Tx) error {
if _, err := tx.CreateBucketIfNotExists([]byte(config.Bucket)); err != nil {
return err
}
return nil
}); err != nil {
return nil, fmt.Errorf("can't create bbolt bucket %q: %w", config.Bucket, err)
}
result := &Store{
bdb: bdb,
bucket: []byte(config.Bucket),
}
go result.cleanupThread(ctx)
return result, nil
}
func (Factory) Valid(data json.RawMessage) error {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
return nil
}
type Config struct {
Path string `json:"path"`
Bucket string `json:"bucket,omitempty"`
}
func (c Config) Valid() error {
var errs []error
if c.Path == "" {
errs = append(errs, ErrMissingPath)
} else {
dir := filepath.Dir(c.Path)
if err := os.WriteFile(filepath.Join(dir, ".test-file"), []byte(""), 0600); err != nil {
errs = append(errs, ErrCantWriteToPath)
}
}
if len(errs) != 0 {
return errors.Join(errs...)
}
return nil
}

View File

@@ -0,0 +1,42 @@
package bbolt
import (
"encoding/json"
"errors"
"testing"
)
func TestFactoryValid(t *testing.T) {
f := Factory{}
t.Run("bad config", func(t *testing.T) {
if err := f.Valid(json.RawMessage(`}`)); err == nil {
t.Error("wanted parsing failure but got a successful result")
}
})
t.Run("invalid config", func(t *testing.T) {
for _, tt := range []struct {
name string
cfg Config
err error
}{
{
name: "missing path",
cfg: Config{},
err: ErrMissingPath,
},
} {
t.Run(tt.name, func(t *testing.T) {
data, err := json.Marshal(tt.cfg)
if err != nil {
t.Fatal(err)
}
if err := f.Valid(json.RawMessage(data)); !errors.Is(err, tt.err) {
t.Error(err)
}
})
}
})
}

90
lib/store/interface.go Normal file
View File

@@ -0,0 +1,90 @@
package store
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
)
var (
// ErrNotFound is returned when the store implementation cannot find the value
// for a given key.
ErrNotFound = errors.New("store: key not found")
// ErrCantDecode is returned when a store adaptor cannot decode the store format
// to a value used by the code.
ErrCantDecode = errors.New("store: can't decode value")
// ErrCantEncode is returned when a store adaptor cannot encode the value into
// the format that the store uses.
ErrCantEncode = errors.New("store: can't encode value")
// ErrBadConfig is returned when a store adaptor's configuration is invalid.
ErrBadConfig = errors.New("store: configuration is invalid")
)
// Interface defines the calls that Anubis uses for storage in a local or remote
// datastore. This can be implemented with an in-memory, on-disk, or in-database
// storage backend.
type Interface interface {
// Delete removes a value from the store by key.
Delete(ctx context.Context, key string) error
// Get returns the value of a key assuming that value exists and has not expired.
Get(ctx context.Context, key string) ([]byte, error)
// Set puts a value into the store that expires according to its expiry.
Set(ctx context.Context, key string, value []byte, expiry time.Duration) error
}
func z[T any]() T { return *new(T) }
type JSON[T any] struct {
Underlying Interface
Prefix string
}
func (j *JSON[T]) Delete(ctx context.Context, key string) error {
if j.Prefix != "" {
key = j.Prefix + key
}
return j.Underlying.Delete(ctx, key)
}
func (j *JSON[T]) Get(ctx context.Context, key string) (T, error) {
if j.Prefix != "" {
key = j.Prefix + key
}
data, err := j.Underlying.Get(ctx, key)
if err != nil {
return z[T](), err
}
var result T
if err := json.Unmarshal(data, &result); err != nil {
return z[T](), fmt.Errorf("%w: %w", ErrCantDecode, err)
}
return result, nil
}
func (j *JSON[T]) Set(ctx context.Context, key string, value T, expiry time.Duration) error {
if j.Prefix != "" {
key = j.Prefix + key
}
data, err := json.Marshal(value)
if err != nil {
return fmt.Errorf("%w: %w", ErrCantEncode, err)
}
if err := j.Underlying.Set(ctx, key, data, expiry); err != nil {
return err
}
return nil
}

50
lib/store/json_test.go Normal file
View File

@@ -0,0 +1,50 @@
package store_test
import (
"testing"
"time"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestJSON(t *testing.T) {
type data struct {
ID string `json:"id"`
}
st := memory.New(t.Context())
db := store.JSON[data]{
Underlying: st,
Prefix: "foo:",
}
if err := db.Set(t.Context(), "test", data{ID: t.Name()}, time.Minute); err != nil {
t.Fatal(err)
}
got, err := db.Get(t.Context(), "test")
if err != nil {
t.Fatal(err)
}
if got.ID != t.Name() {
t.Fatalf("got wrong data for key \"test\", wanted %q but got: %q", t.Name(), got.ID)
}
if err := db.Delete(t.Context(), "test"); err != nil {
t.Fatal(err)
}
if _, err := db.Get(t.Context(), "test"); err == nil {
t.Fatal("wanted invalid get to fail, it did not")
}
if err := st.Set(t.Context(), "foo:test", []byte("}"), time.Minute); err != nil {
t.Fatal(err)
}
if _, err := db.Get(t.Context(), "test"); err == nil {
t.Fatal("wanted invalid get to fail, it did not")
}
}

View File

@@ -0,0 +1,74 @@
package memory
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/lib/store"
)
type factory struct{}
func (factory) Build(ctx context.Context, _ json.RawMessage) (store.Interface, error) {
return New(ctx), nil
}
func (factory) Valid(json.RawMessage) error { return nil }
func init() {
store.Register("memory", factory{})
}
type impl struct {
store *decaymap.Impl[string, []byte]
}
func (i *impl) Delete(_ context.Context, key string) error {
if !i.store.Delete(key) {
return fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return nil
}
func (i *impl) Get(_ context.Context, key string) ([]byte, error) {
result, ok := i.store.Get(key)
if !ok {
return nil, fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return result, nil
}
func (i *impl) Set(_ context.Context, key string, value []byte, expiry time.Duration) error {
i.store.Set(key, value, expiry)
return nil
}
func (i *impl) cleanupThread(ctx context.Context) {
t := time.NewTicker(5 * time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
i.store.Cleanup()
}
}
}
// New creates a simple in-memory store. This will not scale to multiple Anubis instances.
func New(ctx context.Context) store.Interface {
result := &impl{
store: decaymap.New[string, []byte](),
}
go result.cleanupThread(ctx)
return result
}

View File

@@ -0,0 +1,11 @@
package memory
import (
"testing"
"github.com/TecharoHQ/anubis/lib/store/storetest"
)
func TestImpl(t *testing.T) {
storetest.Common(t, factory{}, nil)
}

43
lib/store/registry.go Normal file
View File

@@ -0,0 +1,43 @@
package store
import (
"context"
"encoding/json"
"sort"
"sync"
)
var (
registry map[string]Factory = map[string]Factory{}
regLock sync.RWMutex
)
type Factory interface {
Build(ctx context.Context, config json.RawMessage) (Interface, error)
Valid(config json.RawMessage) error
}
func Register(name string, impl Factory) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Factory, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}

View File

@@ -0,0 +1,92 @@
package storetest
import (
"bytes"
"encoding/json"
"errors"
"testing"
"time"
"github.com/TecharoHQ/anubis/lib/store"
)
func Common(t *testing.T, f store.Factory, config json.RawMessage) {
if err := f.Valid(config); err != nil {
t.Fatal(err)
}
s, err := f.Build(t.Context(), config)
if err != nil {
t.Fatal(err)
}
for _, tt := range []struct {
name string
doer func(t *testing.T, s store.Interface) error
err error
}{
{
name: "basic get set delete",
doer: func(t *testing.T, s store.Interface) error {
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to not exist in store but it exists anyways", t.Name())
}
if err := s.Set(t.Context(), t.Name(), []byte(t.Name()), 5*time.Minute); err != nil {
return err
}
val, err := s.Get(t.Context(), t.Name())
if errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to exist in store but it does not", t.Name())
}
if !bytes.Equal(val, []byte(t.Name())) {
t.Logf("want: %q", t.Name())
t.Logf("got: %q", string(val))
t.Error("wrong value returned")
}
if err := s.Delete(t.Context(), t.Name()); err != nil {
return err
}
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Error("wanted test to not exist in store but it exists anyways")
}
if err := s.Delete(t.Context(), t.Name()); err == nil {
t.Errorf("key %q does not exist and Delete did not return non-nil", t.Name())
}
return nil
},
},
{
name: "expires",
doer: func(t *testing.T, s store.Interface) error {
if err := s.Set(t.Context(), t.Name(), []byte(t.Name()), 150*time.Millisecond); err != nil {
return err
}
//nosleep:bypass XXX(Xe): use Go's time faking thing in Go 1.25 when that is released.
time.Sleep(155 * time.Millisecond)
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to not exist in store but it exists anyways", t.Name())
}
return nil
},
},
} {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if err := tt.doer(t, s); !errors.Is(err, tt.err) {
t.Logf("want: %v", tt.err)
t.Logf("got: %v", err)
t.Error("wrong error")
}
})
}
}

View File

@@ -0,0 +1,84 @@
package valkey
import (
"context"
"encoding/json"
"errors"
"fmt"
"github.com/TecharoHQ/anubis/lib/store"
valkey "github.com/redis/go-redis/v9"
)
var (
ErrNoURL = errors.New("valkey.Config: no URL defined")
ErrBadURL = errors.New("valkey.Config: URL is invalid")
)
func init() {
store.Register("valkey", Factory{})
}
type Factory struct{}
func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface, error) {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
opts, err := valkey.ParseURL(config.URL)
if err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
rdb := valkey.NewClient(opts)
if _, err := rdb.Ping(ctx).Result(); err != nil {
return nil, fmt.Errorf("can't ping valkey instance: %w", err)
}
return &Store{
rdb: rdb,
}, nil
}
func (Factory) Valid(data json.RawMessage) error {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
return nil
}
type Config struct {
URL string `json:"url"`
}
func (c Config) Valid() error {
var errs []error
if c.URL == "" {
errs = append(errs, ErrNoURL)
}
if _, err := valkey.ParseURL(c.URL); err != nil {
errs = append(errs, ErrBadURL)
}
if len(errs) != 0 {
return fmt.Errorf("valkey.Config: invalid config: %w", errors.Join(errs...))
}
return nil
}

View File

@@ -0,0 +1,49 @@
package valkey
import (
"context"
"fmt"
"time"
"github.com/TecharoHQ/anubis/lib/store"
valkey "github.com/redis/go-redis/v9"
)
type Store struct {
rdb *valkey.Client
}
func (s *Store) Delete(ctx context.Context, key string) error {
n, err := s.rdb.Del(ctx, key).Result()
if err != nil {
return fmt.Errorf("can't delete from valkey: %w", err)
}
switch n {
case 0:
return fmt.Errorf("%w: %d key(s) deleted", store.ErrNotFound, n)
default:
return nil
}
}
func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
result, err := s.rdb.Get(ctx, key).Result()
if err != nil {
if valkey.HasErrorPrefix(err, "redis: nil") {
return nil, fmt.Errorf("%w: %w", store.ErrNotFound, err)
}
return nil, fmt.Errorf("can't fetch from valkey: %w", err)
}
return []byte(result), nil
}
func (s *Store) Set(ctx context.Context, key string, value []byte, expiry time.Duration) error {
if _, err := s.rdb.Set(ctx, key, string(value), expiry).Result(); err != nil {
return fmt.Errorf("can't set %q in valkey: %w", key, err)
}
return nil
}

View File

@@ -0,0 +1,53 @@
package valkey
import (
"encoding/json"
"fmt"
"os"
"testing"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/lib/store/storetest"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)
func init() {
internal.UnbreakDocker()
}
func TestImpl(t *testing.T) {
if os.Getenv("DONT_USE_NETWORK") != "" {
t.Skip("test requires network egress")
return
}
testcontainers.SkipIfProviderIsNotHealthy(t)
req := testcontainers.ContainerRequest{
Image: "valkey/valkey:8",
WaitingFor: wait.ForLog("Ready to accept connections"),
}
valkeyC, err := testcontainers.GenericContainer(t.Context(), testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
})
testcontainers.CleanupContainer(t, valkeyC)
if err != nil {
t.Fatal(err)
}
containerIP, err := valkeyC.ContainerIP(t.Context())
if err != nil {
t.Fatal(err)
}
data, err := json.Marshal(Config{
URL: fmt.Sprintf("redis://%s:6379/0", containerIP),
})
if err != nil {
t.Fatal(err)
}
storetest.Common(t, Factory{}, json.RawMessage(data))
}