Compare commits

...

23 Commits

Author SHA1 Message Date
Xe Iaso
be42c5accf fix(lib/store/bbolt): run cleanup every hour instead of every 5 minutes
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-06 01:15:11 +00:00
Xe Iaso
7d0c58d1a8 fix: make ogtags and dnsbl use the Store instead of memory (#760)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:17:46 -04:00
Xe Iaso
e870ede120 docs(known-instances): add git.aya.so
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:41:36 +00:00
Xe Iaso
592d1e3dfc docs(known-instances): add Pluralpedia
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 16:40:32 +00:00
Xe Iaso
f6254b4b98 docs(installation): clarify BASE_PREFIX matches the /.within.website endpoints
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 13:47:02 +00:00
Lothar Serra Mari
d19026d693 docs(known-instances): Add Duke University, coinhoards.org (and myself) to known instances (#757)
* docs(known-instances): add Duke University to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* docs(known-instances): add fabulous.systems to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* docs(known-instances): add coinhoards.org to known instances

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* chore(spelling): exempt the known instances page

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Lothar Serra Mari <mail@serra.me>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-07-05 08:29:44 -04:00
Xe Iaso
7b72c790ab chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 12:29:19 +00:00
Xe Iaso
719a1409ca test(lib/store/bbolt): disable this test case for now
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 04:56:19 +00:00
Xe Iaso
890f21bf47 chore(devcontainer): move playwright to its own devcontainer service (#756)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-05 00:53:45 -04:00
Xe Iaso
93bfe910d8 docs(user/faq): clarify Anubis not being a cryptocurrency miner
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 22:49:39 +00:00
Xe Iaso
19d8de784b chore(docs/manifest): enable bbolt in an emptyDir
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 21:07:37 +00:00
Xe Iaso
dff2176beb feat(lib): use new challenge creation flow (#749)
* feat(decaymap): add Delete method

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(lib/challenge): refactor Validate to take ValidateInput

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib): implement store interface

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): all metapackage to import all store implementations

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(policy): import all store backends

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib): use new challenge creation flow

Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.

This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(decaymap): fix documentation typo

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(lib): fix SA4004

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/store): make generic storage interface test adaptor

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(decaymap): invert locking process for Delete

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): add bbolt store implementation

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: go mod tidy

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(devcontainer): adapt to docker compose, add valkey service

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): make challenges live for 30 minutes by default

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/store): implement valkey backend

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/store/valkey): disable tests if not using docker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/policy/config): ensure valkey stores can be loaded

Signed-off-by: Xe Iaso <me@xeiaso.net>

* Update metadata

check-spelling run (pull_request) for Xe/store-interface

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* chore(devcontainer): remove port forwards because vs code handles that for you

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(default-config): add a nudge to the storage backends section of the docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(docs): listen on 0.0.0.0 for dev container support

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(policy): document storage backends

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG and internal links

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(admin/policies): don't start a sentence with as

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: fixes found in review

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
2025-07-04 20:42:28 +00:00
SGHFan
506d8817d5 Update known-instances.md (#755)
Add eBird to instance list

Signed-off-by: SGHFan <62897482+SGHFan@users.noreply.github.com>
2025-07-04 19:54:26 +00:00
Duru Can Celasun
d0fae02d05 feat(localization): Add Turkish language translation (#751)
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-07-04 00:50:16 -04:00
Xe Iaso
845095c3f6 chore(robots.txt): don't block CCBot
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 00:23:21 +00:00
Xe Iaso
2f1e78cc6c chore(docs/manifest): allow common crawl to test with the team
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 00:22:43 +00:00
Xe Iaso
7c0996448a chore(default-config): allowlist common crawl (#753)
This may seem strange, but allowlisting common crawl means that scrapers
have less incentive to scrape because they can just grab the data from
common crawl instead of scraping it again.
2025-07-04 00:10:45 +00:00
Xe Iaso
d7a758f805 docs: add BotStopper docs from the git repo (#752)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-03 23:09:45 +00:00
Martin
c121896f9c feat(localization): Add German language translation (#741)
* Add german translation

* Adjust german localization

* Adjust js_finished_reading in german localization

* Mention this change in CHANGELOG.md

* Add test for German localization

* Update lib/localization/locales/de.json

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
Signed-off-by: Martin <31348196+Earl0fPudding@users.noreply.github.com>

* Remove duplicate "leider" in lib/localization/locales/de.json

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
Signed-off-by: Martin <31348196+Earl0fPudding@users.noreply.github.com>

* Update lib/localization/locales/de.json

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
Signed-off-by: Martin <31348196+Earl0fPudding@users.noreply.github.com>

* Update lib/localization/locales/de.json

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
Signed-off-by: Martin <31348196+Earl0fPudding@users.noreply.github.com>

---------

Signed-off-by: Martin <31348196+Earl0fPudding@users.noreply.github.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-07-03 10:48:17 +00:00
Xe Iaso
888b7d6e77 fix(run/anubis@.service): unique runtimedir per instance (#750)
* fix(run/anubis@.service): unique runtimedir per instance

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-03 10:29:05 +00:00
Martin
0e43138324 feat(localization): Add option for forcing a language (#742)
* Add forcesLanguage option

* Change comments for forced language option

* Add changes to CHANGELOG.md

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-07-02 05:33:00 +00:00
Xe Iaso
c981c23f7e chore: npm run generate
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-02 05:25:10 +00:00
Xe Iaso
9f0c5e974e fix(web/main): remove the success interstitial (#745)
I'm gonna be totally honest here, I'm still not sure why #564 is still
an issue. This is really confusing and I'm going to totally throw out
how Anubis issues challenges and redo it with Valkey (#201, #622).

The problem seems to be that I assume that the makeChallenge function in
package lib is idempotent for the same client. I have no idea why this
would be inconsistent, but for some reason it is and I'm just at a loss
for words as to why this is happening.

This stops the bleeding by improving the UX as a stopgap.

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-01 23:44:38 +00:00
81 changed files with 2144 additions and 319 deletions

View File

@@ -3,10 +3,10 @@ FROM ghcr.io/xe/devcontainer-base/pre/go
WORKDIR /app
COPY go.mod go.sum package.json package-lock.json ./
RUN go install github.com/a-h/templ/cmd/templ \
&& npx --yes playwright@1.52.0 install --with-deps\
&& apt-get update \
&& apt-get -y install zstd brotli \
RUN apt-get update \
&& apt-get -y install zstd brotli redis \
&& mkdir -p /home/vscode/.local/share/fish \
&& chown -R vscode:vscode /home/vscode/.local/share/fish \
&& chown -R vscode:vscode /go
&& chown -R vscode:vscode /go
CMD ["/usr/bin/sleep", "infinity"]

View File

@@ -2,14 +2,9 @@
// README at: https://github.com/devcontainers/templates/tree/main/src/debian
{
"name": "Dev",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"build": {
"dockerfile": "./Dockerfile",
"context": "..",
"cacheFrom": [
"type=registry,ref=ghcr.io/techarohq/anubis/devcontainer"
]
},
"dockerComposeFile": ["./docker-compose.yaml"],
"service": "workspace",
"workspaceFolder": "/workspace/anubis",
"postStartCommand": "npm ci && go mod download",
"features": {
"ghcr.io/xe/devcontainer-features/ko:1.1.0": {}
@@ -26,9 +21,5 @@
"redhat.vscode-yaml"
]
}
},
"forwardPorts": [
8923,
3000
]
}
}
}

View File

@@ -0,0 +1,26 @@
services:
playwright:
image: mcr.microsoft.com/playwright:v1.52.0-noble
init: true
network_mode: service:workspace
command:
- /bin/sh
- -c
- npx -y playwright@1.52.0 run-server --port 9001 --host 0.0.0.0
valkey:
image: valkey/valkey:8
pull_policy: always
# VS Code workspace service
workspace:
image: ghcr.io/techarohq/anubis/devcontainer
build:
context: ..
dockerfile: .devcontainer/Dockerfile
volumes:
- ../:/workspace/anubis:cached
environment:
VALKEY_URL: redis://valkey:6379/0
#entrypoint: ["/usr/bin/sleep", "infinity"]
user: vscode

View File

@@ -84,6 +84,7 @@
^\Q.github/workflows/spelling.yml\E$
^data/crawlers/
^docs/blog/tags\.yml$
^docs/docs/user/known-instances.md$
^docs/manifest/.*$
^docs/static/\.nojekyll$
^lib/policy/config/testdata/bad/unparseable\.json$

View File

@@ -16,15 +16,18 @@ aspirational
atuin
azuretools
badregexes
bbolt
bdba
berr
bingbot
bitcoin
Bitcoin
bitrate
blogging
Bluesky
blueskybot
boi
botnet
botstopper
BPort
Brightbot
broked
@@ -44,14 +47,17 @@ cgr
chainguard
chall
challengemozilla
challengetest
checkpath
checkresult
chibi
cidranger
ckie
ckies
cloudflare
Codespaces
confd
connnection
containerbuild
coreutils
Cotoyogi
@@ -75,6 +81,7 @@ domainhere
dracula
dronebl
droneblresponse
dropin
duckduckbot
eerror
ellenjoe
@@ -92,6 +99,7 @@ facebookgo
Factset
fastcgi
fediverse
ffprobe
finfos
Firecrawl
flagenv
@@ -122,7 +130,6 @@ Hashcash
hashrate
headermap
healthcheck
hebis
hec
hmc
hostable
@@ -198,6 +205,7 @@ omgilibot
openai
opengraph
openrc
oswald
pag
palemoon
Pangu
@@ -226,6 +234,7 @@ qwantbot
rac
rawler
rcvar
rdb
redhat
redir
redirectscheme
@@ -237,9 +246,9 @@ risc
ruleset
runlevels
RUnlock
runtimedir
sas
sasl
Scumm
searchbot
searx
sebest
@@ -252,10 +261,8 @@ shellcheck
Sidetrade
simprint
sitemap
skopeo
sls
sni
Sourceware
Spambot
sparkline
spyderbot
@@ -263,6 +270,7 @@ srv
stackoverflow
startprecmd
stoppostcmd
storetest
subgrid
subr
subrequest
@@ -270,6 +278,8 @@ SVCNAME
tagline
tarballs
tarrif
tbn
tbr
techaro
techarohq
templ
@@ -282,6 +292,8 @@ Tik
Timpibot
traefik
uberspace
Unbreak
unbreakdocker
unifiedjs
unixhttpd
unmarshal
@@ -289,7 +301,7 @@ unparseable
uuidgen
uvx
UXP
Valkey
valkey
Varis
Velen
vendored
@@ -328,5 +340,4 @@ yoursite
Zenos
zizmor
zombocom
Zonbocom
zos

View File

@@ -1,47 +0,0 @@
name: Dev container prebuild
on:
push:
branches: ["main"]
tags: ["v*.*.*"]
jobs:
devcontainer:
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-tags: true
fetch-depth: 0
persist-credentials: false
- name: Set up QEMU
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: latest
- run: |
sudo apt-get update
sudo apt-get -y install skopeo
- name: Log into registry
if: github.event_name != 'pull_request'
uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3.0.0
with:
registry: ghcr.io
username: techarohq
password: ${{ secrets.GITHUB_TOKEN }}
- name: Pre-build dev container image
uses: devcontainers/ci@8bf61b26e9c3a98f69cb6ce2f88d24ff59b785c6 # v0.3.1900000417
with:
imageName: ghcr.io/techarohq/anubis/devcontainer
cacheFrom: ghcr.io/techarohq/anubis/devcontainer
push: always
platform: linux/amd64,linux/arm64

View File

@@ -32,3 +32,7 @@ const APIPrefix = "/.within.website/x/cmd/anubis/api/"
// DefaultDifficulty is the default "difficulty" (number of leading zeroes)
// that must be met by the client in order to pass the challenge.
const DefaultDifficulty = 4
// ForcedLanguage is the language being used instead of the one of the request's Accept-Language header
// if being set.
var ForcedLanguage = ""

View File

@@ -50,6 +50,7 @@ var (
cookieExpiration = flag.Duration("cookie-expiration-time", anubis.CookieDefaultExpirationTime, "The amount of time the authorization cookie is valid for")
cookiePrefix = flag.String("cookie-prefix", "techaro.lol-anubis", "prefix for browser cookies created by Anubis")
cookiePartitioned = flag.Bool("cookie-partitioned", false, "if true, sets the partitioned flag on Anubis cookies, enabling CHIPS support")
forcedLanguage = flag.String("forced-language", "", "if set, this language is being used instead of the one from the request's Accept-Language header")
hs512Secret = flag.String("hs512-secret", "", "secret used to sign JWTs, uses ed25519 if not set")
cookieSecure = flag.Bool("cookie-secure", true, "if true, sets the secure flag on Anubis cookies")
ed25519PrivateKeyHex = flag.String("ed25519-private-key-hex", "", "private key used to sign JWTs, if not set a random one will be assigned")
@@ -230,20 +231,6 @@ func makeReverseProxy(target string, targetSNI string, targetHost string, insecu
return rp, nil
}
func startDecayMapCleanup(ctx context.Context, s *libanubis.Server) {
ticker := time.NewTicker(1 * time.Hour)
defer ticker.Stop()
for {
select {
case <-ticker.C:
s.CleanupDecayMap()
case <-ctx.Done():
return
}
}
}
func main() {
flagenv.Parse()
flag.Parse()
@@ -378,6 +365,7 @@ func main() {
anubis.CookieName = *cookiePrefix + "-auth"
anubis.TestCookieName = *cookiePrefix + "-cookie-verification"
anubis.ForcedLanguage = *forcedLanguage
// If OpenGraph configuration values are not set in the config file, use the
// values from flags / envvars.
@@ -419,7 +407,6 @@ func main() {
wg.Add(1)
go metricsServer(ctx, wg.Done)
}
go startDecayMapCleanup(ctx, s)
var h http.Handler
h = s

View File

@@ -145,6 +145,14 @@ status_codes:
CHALLENGE: 200
DENY: 200
# Anubis can store temporary data in one of a few backends. See the storage
# backends section of the docs for more information:
#
# https://anubis.techaro.lol/docs/admin/policies#storage-backends
store:
backend: memory
parameters: {}
# The weight thresholds for when to trigger individual challenges. Any
# CHALLENGE will take precedence over this.
#

View File

@@ -7,5 +7,5 @@
# Warning: May contain user agents that _must_ be blocked in robots.txt, or the opt-out will have no effect.
- name: "ai-catchall"
user_agent_regex: >-
AI2Bot|Ai2Bot-Dolma|aiHitBot|Amazonbot|anthropic-ai|Brightbot 1.0|Bytespider|CCBot|Claude-Web|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google-CloudVertexBot|GoogleOther|GoogleOther-Image|GoogleOther-Video|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo Bot|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|NovaAct|omgili|omgilibot|Operator|PanguBot|Perplexity-User|PerplexityBot|PetalBot|QualifiedBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio-Extended|wpbot|YouBot
AI2Bot|Ai2Bot-Dolma|aiHitBot|Amazonbot|anthropic-ai|Brightbot 1.0|Bytespider|Claude-Web|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google-CloudVertexBot|GoogleOther|GoogleOther-Image|GoogleOther-Video|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo Bot|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|NovaAct|omgili|omgilibot|Operator|PanguBot|Perplexity-User|PerplexityBot|PetalBot|QualifiedBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio-Extended|wpbot|YouBot
action: DENY

View File

@@ -1,6 +1,8 @@
# Warning: Contains user agents that _must_ be blocked in robots.txt, or the opt-out will have no effect.
# Note: Blocks human-directed/non-training user agents
#
# CCBot is allowed because if Common Crawl is allowed, then scrapers don't need to scrape to get the data.
- name: "ai-robots-txt"
user_agent_regex: >-
AI2Bot|Ai2Bot-Dolma|aiHitBot|Amazonbot|Andibot|anthropic-ai|Applebot|Applebot-Extended|bedrockbot|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-SearchBot|Claude-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|EchoboxBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google-CloudVertexBot|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo Bot|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|MistralAI-User/1.0|MyCentralAIScraperBot|NovaAct|OAI-SearchBot|omgili|omgilibot|Operator|PanguBot|Panscient|panscient.com|Perplexity-User|PerplexityBot|PetalBot|PhindBot|Poseidon Research Crawler|QualifiedBot|QuillBot|quillbot.com|SBIntuitionsBot|Scrapy|SemrushBot|SemrushBot-BA|SemrushBot-CT|SemrushBot-OCOB|SemrushBot-SI|SemrushBot-SWA|Sidetrade indexer bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio-Extended|wpbot|YandexAdditional|YandexAdditionalBot|YouBot
AI2Bot|Ai2Bot-Dolma|aiHitBot|Amazonbot|Andibot|anthropic-ai|Applebot|Applebot-Extended|bedrockbot|Brightbot 1.0|Bytespider|ChatGPT-User|Claude-SearchBot|Claude-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|EchoboxBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google-CloudVertexBot|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo Bot|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|MistralAI-User/1.0|MyCentralAIScraperBot|NovaAct|OAI-SearchBot|omgili|omgilibot|Operator|PanguBot|Panscient|panscient.com|Perplexity-User|PerplexityBot|PetalBot|PhindBot|Poseidon Research Crawler|QualifiedBot|QuillBot|quillbot.com|SBIntuitionsBot|Scrapy|SemrushBot|SemrushBot-BA|SemrushBot-CT|SemrushBot-OCOB|SemrushBot-SI|SemrushBot-SWA|Sidetrade indexer bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio-Extended|wpbot|YandexAdditional|YandexAdditionalBot|YouBot
action: DENY

View File

@@ -6,4 +6,5 @@
- import: (data)/crawlers/internet-archive.yaml
- import: (data)/crawlers/kagibot.yaml
- import: (data)/crawlers/marginalia.yaml
- import: (data)/crawlers/mojeekbot.yaml
- import: (data)/crawlers/mojeekbot.yaml
- import: (data)/crawlers/commoncrawl.yaml

View File

@@ -0,0 +1,12 @@
- name: common-crawl
user_agent_regex: CCBot
action: ALLOW
# https://index.commoncrawl.org/ccbot.json
remote_addresses:
[
"2600:1f28:365:80b0::/60",
"18.97.9.168/29",
"18.97.14.80/29",
"18.97.14.88/30",
"98.85.178.216/32",
]

View File

@@ -48,6 +48,26 @@ func (m *Impl[K, V]) expire(key K) bool {
return true
}
// Delete a value from the DecayMap by key.
//
// If the value does not exist, return false. Return true after
// deletion.
func (m *Impl[K, V]) Delete(key K) bool {
m.lock.RLock()
_, ok := m.data[key]
m.lock.RUnlock()
if !ok {
return false
}
m.lock.Lock()
delete(m.data, key)
m.lock.Unlock()
return true
}
// Get gets a value from the DecayMap by key.
//
// If a value has expired, forcibly delete it if it was not updated.

View File

@@ -10,15 +10,41 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
<!-- This changes the project to: -->
- Add `COOKIE_SECURE` option to set the cookie [Secure flag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies)
- Sets cookie defaults to use [SameSite: None](https://web.dev/articles/samesite-cookies-explained)
- Determine the `BIND_NETWORK`/`--bind-network` value from the bind address ([#677](https://github.com/TecharoHQ/anubis/issues/677)).
- Implement localization system. Find locale files in lib/localization/locales/.
- Implement a [development container](https://containers.dev/) manifest to make contributions easier.
- Fix dynamic cookie domains functionality ([#731](https://github.com/TecharoHQ/anubis/pull/731))
- Add option for custom cookie prefix ([#732](https://github.com/TecharoHQ/anubis/pull/732))
- Add translation for German language ([#741](https://github.com/TecharoHQ/anubis/pull/741))
- Remove the "Success" interstitial after a proof of work challenge is concluded.
- Anubis now has the concept of [storage backends](./admin/policies.mdx#storage-backends). These allow you to change how Anubis stores temporary data (in memory, on the disk, or in Valkey). If you run Anubis in an environment where you have a low amount of memory available for Anubis (eg: less than 64 megabytes), be sure to configure the [`bbolt`](./admin/policies.mdx#bbolt) storage backend.
- The challenge issuance and validation process has been rewritten from scratch. Instead of generating challenge strings from request metadata (under the assumption that the values being compared against are stable), Anubis now generates random data for each challenge. This data is stored in the active [storage backend](./admin/policies.mdx#storage-backends) for up to 30 minutes. Fixes [#564](https://github.com/TecharoHQ/anubis/issues/564), [#746](https://github.com/TecharoHQ/anubis/issues/746), and other similar instances of this issue.
- Make the [Open Graph](./admin/configuration/open-graph.mdx) subsystem and DNSBL subsystem use [storage backends](./admin/policies.mdx#storage-backends) instead of storing everything in memory by default.
- Add option for forcing a specific language ([#742](https://github.com/TecharoHQ/anubis/pull/742))
- Add translation for Turkish language ([#751](https://github.com/TecharoHQ/anubis/pull/751))
- Allow [Common Crawl](https://commoncrawl.org/) by default so scrapers have less incentive to scrape
- The [bbolt storage backend](./admin/policies.mdx#bbolt) now runs its cleanup every hour instead of every five minutes.
### Potentially breaking changes
The following potentially breaking change applies to native installs with systemd only:
Each instance of systemd service template now has a unique `RuntimeDirectory`, as opposed to each instance of the service sharing a `RuntimeDirectory`. This change was made to avoid [the `RuntimeDirectory` getting nuked any time one of the Anubis instances restarts](https://github.com/TecharoHQ/anubis/issues/748).
If you configured Anubis' unix sockets to listen on `/run/anubis/foo.sock` for instance `anubis@foo`, you will need to configure Anubis to listen on `/run/anubis/foo/sock` and additionally configure your HTTP load balancer as appropriate.
If you need the legacy behaviour, install this [systemd unit dropin](https://www.flatcar.org/docs/latest/setup/systemd/drop-in-units/):
```systemd
# /etc/systemd/system/anubis@.service.d/50-runtimedir.conf
[Service]
RuntimeDirectory=anubis
```
## v1.20.0: Thancred Waters

View File

@@ -0,0 +1,215 @@
---
title: "Commercial support and an unbranded version"
---
If you want to use Anubis but organizational policies prevent you from using the branding that the open source project ships, we offer a commercial version of Anubis named BotStopper. BotStopper builds off of the open source core of Anubis and offers organizations more control over the branding, including but not limited to:
- Custom images for different states of the challenge process (in process, success, failure)
- Custom CSS and fonts
- Custom titles for the challenge and error pages
- "Anubis" replaced with "BotStopper" across the UI
- A private bug tracker for issues
In the near future this will expand to:
- A private challenge implementation that does advanced fingerprinting to check if the client is a genuine browser or not
- Advanced fingerprinting via [Thoth-based advanced checks](./thoth.mdx)
In order to sign up for BotStopper, please do one of the following:
- Sign up [on GitHub Sponsors](https://github.com/sponsors/Xe) at the $50 per month tier or higher
- Email [sales@techaro.lol](mailto:sales@techaro.lol) with your requirements for invoicing, please note that custom invoicing will cost more than using GitHub Sponsors for understandable overhead reasons
## Installation
Install BotStopper like you would Anubis, but replace the image reference. EG:
```diff
-ghcr.io/techarohq/anubis:latest
+ghcr.io/techarohq/botstopper/anubis:latest
```
### Binary packages
Binary packages are available [in the GitHub Releases page](https://github.com/TecharoHQ/botstopper/releases), the main difference is that the package name is `techaro-botstopper`, the systemd service is `techaro-botstopper@your-instance.service`, the binary is `/usr/bin/botstopper`, and the configuration is in `/etc/techaro-botstopper`. All other instructions in the [native package install guide](./native-install.mdx) apply.
### Docker / Podman
In order to pull the BotStopper image, you need to [authenticate with GitHub's Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry).
```text
docker login ghcr.io -u your-username --password-stdin
```
Then you can use the image as normal.
### Kubernetes
If you are using Kubernetes, you will need to create an image pull secret:
```text
kubectl create secret docker-registry \
techarohq-botstopper \
--docker-server ghcr.io \
--docker-username your-username \
--docker-password your-access-token \
--docker-email your@email.address
```
Then attach it to your Deployment:
```diff
spec:
securityContext:
fsGroup: 1000
+ imagePullSecrets:
+ - name: techarohq-botstopper
```
## Configuration
### Docker compose
Follow [the upstream Docker compose directions](https://anubis.techaro.lol/docs/admin/environments/docker-compose) with the following additional options:
```diff
anubis:
image: ghcr.io/techarohq/botstopper/anubis:latest
environment:
BIND: ":8080"
DIFFICULTY: "4"
METRICS_BIND: ":9090"
SERVE_ROBOTS_TXT: "true"
TARGET: "http://nginx"
OG_PASSTHROUGH: "true"
OG_EXPIRY_TIME: "24h"
+ # botstopper config here
+ CHALLENGE_TITLE: "Doing math for your connnection!"
+ ERROR_TITLE: "Something went wrong!"
+ OVERLAY_FOLDER: /assets
+ volumes:
+ - "./your_folder:/assets"
```
#### Example
There is an example in [docker-compose.yaml](https://github.com/TecharoHQ/botstopper/blob/main/docker-compose.yaml). Start the example with `docker compose up`:
```text
docker compose up -d
```
And then open [https://botstopper.local.cetacean.club:8443](https://botstopper.local.cetacean.club:8443) in your browser.
> [!NOTE]
> This uses locally signed sacrificial TLS certificates stored in `./demo/pki`. Your browser will rightly reject these. Here is what the example looks like:
>
> ![](/img/botstopper/example-screenshot.webp)
## Custom images and CSS
Anubis uses an internal filesystem that contains CSS, JavaScript, and images. The BotStopper variant of Anubis lets you specify an overlay folder with the environment variable `OVERLAY_FOLDER`. The contents of this folder will be overlaid on top of Anubis' internal filesystem, allowing you to easily customize the images and CSS.
Your directory tree should look like this, assuming your data is in `./your_folder`:
```text
./your_folder
└── static
├── css
│ └── custom.css
└── img
├── happy.webp
├── pensive.webp
└── reject.webp
```
For an example directory tree using some off-the-shelf images the Tango icon set, see the [testdata](https://github.com/TecharoHQ/botstopper/tree/main/testdata/static/img) folder.
### Custom CSS
CSS customization is done mainly with CSS variables. View [the example custom CSS file](https://github.com/TecharoHQ/botstopper/blob/main/testdata/static/css/custom.css) for more information about what can be customized.
### Custom fonts
If you want to add custom fonts, copy the `woff2` files alongside your `custom.css` file and then include them with the [`@font-face` CSS at-rule](https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face):
```css
@font-face {
font-family: "Oswald";
font-style: normal;
font-weight: 200 900;
font-display: swap;
src: url("./fonts/oswald.woff2") format("woff2");
}
```
Then adjust your CSS variables accordingly:
```css
:root {
--body-sans-font: Oswald, sans-serif;
--body-preformatted-font: monospace;
--body-title-font: serif;
}
```
To convert `.ttf` fonts to [Web-optimized woff2 fonts](https://www.w3.org/TR/WOFF2/), use the `woff2_compress` command from the `woff2` or `woff2-tools` package:
```console
$ woff2_compress oswald.ttf
Processing oswald.ttf => oswald.woff2
Compressed 159517 to 70469.
```
Then you can import and use it as normal.
### Customizing images
Anubis uses three images to visually communicate the state of the program. These are:
| Image name | Intended message | Example |
| :------------- | :----------------------------------------------- | :-------------------------------- |
| `happy.webp` | You have passed validation, all is good | ![](/img/botstopper/happy.webp) |
| `pensive.webp` | Checking is running, hold steady until it's done | ![](/img/botstopper/pensive.webp) |
| `reject.webp` | Something went wrong, this is a terminal state | ![](/img/botstopper/reject.webp) |
To make your own images at the optimal quality, use the following ffmpeg command:
```text
ffmpeg -i /path/to/image -vf scale=-1:384 happy.webp
```
`ffprobe` should report something like this on the generated images:
```text
Input #0, webp_pipe, from 'happy.webp':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: webp, none, 25 fps, 25 tbr, 25 tbn
```
In testing 384 by 384 pixels gives the best balance between filesize, quality, and clarity.
```text
$ du -hs *
4.0K happy.webp
12K pensive.webp
8.0K reject.webp
```
## Customizing messages
You can customize messages using the following environment variables:
| Message | Environment variable | Default |
| :------------------- | :------------------- | :----------------------------------------- |
| Challenge page title | `CHALLENGE_TITLE` | `Ensuring the security of your connection` |
| Error page title | `ERROR_TITLE` | `Error` |
For example:
```sh
# /etc/techaro-botstopper/gitea.env
CHALLENGE_TITLE="Wait a moment please!"
ERROR_TITLE="Client error"
```

View File

@@ -60,14 +60,14 @@ Anubis uses these environment variables for configuration:
| Environment Variable | Default value | Explanation |
| :----------------------------- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `BASE_PREFIX` | unset | If set, adds a global prefix to all Anubis endpoints. For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes. |
| `BASE_PREFIX` | unset | If set, adds a global prefix to all Anubis endpoints (everything starting with `/.within.website/x/anubis/`). For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes. |
| `BIND` | `:8923` | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock` |
| `BIND_NETWORK` | `tcp` | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports. |
| `COOKIE_DOMAIN` | unset | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See this [stackoverflow explanation of cookies](https://stackoverflow.com/a/1063760) for more information.<br/><br/>Note that unlike `REDIRECT_DOMAINS`, you should never include a port number in this variable. |
| `COOKIE_DYNAMIC_DOMAIN` | false | If set to true, automatically set cookie domain fields based on the hostname of the request. EG: if you are making a request to `anubis.techaro.lol`, the Anubis cookie will be valid for any subdomain of `techaro.lol`. |
| `COOKIE_EXPIRATION_TIME` | `168h` | The amount of time the authorization cookie is valid for. |
| `COOKIE_PARTITIONED` | `false` | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe. |
| `COOKIE_SECURE` | `true` | If set to `true`, enables the [Secure flag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies), meaning that the cookies will only be transmitted over HTTPS. If Anubis is used in an unsecure context (plain HTTP), this will be need to be set to false |
| `COOKIE_SECURE` | `true` | If set to `true`, enables the [Secure flag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cookies#block_access_to_your_cookies), meaning that the cookies will only be transmitted over HTTPS. If Anubis is used in an unsecure context (plain HTTP), this will be need to be set to false |
| `DIFFICULTY` | `4` | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses. |
| `ED25519_PRIVATE_KEY_HEX` | unset | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details. |
| `ED25519_PRIVATE_KEY_HEX_FILE` | unset | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set. |

View File

@@ -237,6 +237,115 @@ remote_addresses:
Anubis has support for showing imprint / impressum information. This is defined in the `impressum` block of your configuration. See [Imprint / Impressum configuration](./configuration/impressum.mdx) for more information.
## Storage backends
Anubis needs to store temporary data in order to determine if a user is legitimate or not. Administrators should choose a storage backend based on their infrastructure needs. Each backend has its own advantages and disadvantages.
Anubis offers the following storage backends:
- [`memory`](#memory) -- A simple in-memory hashmap
- [`bbolt`](#bbolt) -- An on-disk key/value store backed by [bbolt](https://github.com/etcd-io/bbolt), an embedded key/value database for Go programs
- [`valkey`](#valkey) -- A remote in-memory key/value database backed by [Valkey](https://valkey.io/) (or another database compatible with the [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/) protocol)
If no storage backend is set in the policy file, Anubis will use the [`memory`](#memory) backend by default. This is equivalent to the following in the policy file:
```yaml
store:
backend: memory
parameters: {}
```
### `memory`
The memory backend is an in-memory cache. This backend works best if you don't use multiple instances of Anubis or don't have mutable storage in the environment you're running Anubis in.
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | ✅ Yes |
| Does your service get a lot of traffic? | 🚫 No |
| Do you want to store data persistently when Anubis restarts? | 🚫 No |
| Do you run Anubis without mutable filesystem storage? | ✅ Yes |
The biggest downside is that there is not currently a limit to how much data can be stored in memory. This will be addressed at a later time.
#### Configuration
The memory backend does not require any configuration to use.
### `bbolt`
An on-disk storage layer powered by [bbolt](https://github.com/etcd-io/bbolt), a high performance embedded key/value database used by containerd, etcd, Kubernetes, and NATS. This backend works best if you're running Anubis on a single host and get a lot of traffic.
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | ✅ Yes |
| Does your service get a lot of traffic? | ✅ Yes |
| Do you want to store data persistently when Anubis restarts? | ✅ Yes |
| Do you run Anubis without mutable filesystem storage? | 🚫 No |
When Anubis opens a bbolt database, it takes an exclusive lock on that database. Other instances of Anubis or other tools cannot view the bbolt database while it is locked by another instance of Anubis. If you run multiple instances of Anubis for different services, give each its own `bbolt` configuration.
#### Configuration
The `bbolt` backend takes the following configuration options:
| Name | Type | Example | Description |
| :------- | :----- | :----------------- | :-------------------------------------------------------------------------------------------------------------------------------- |
| `bucket` | string | `anubis` | The bbolt bucket that Anubis should place all its data into. If this is not set, then Anubis will default to the bucket `anubis`. |
| `path` | path | `/data/anubis.bdb` | The filesystem path for the Anubis bbolt database. Anubis requires write access to the folder containing the bbolt database. |
Example:
If you have persistent storage mounted to `/data`, then your store configuration could look like this:
```yaml
store:
backend: bbolt
parameters:
path: /data/anubis.bdb
```
### `valkey`
[Valkey](https://valkey.io/) is an in-memory key/value store that clients access over the network. This allows multiple instances of Anubis to share information and does not require each instance of Anubis to have persistent filesystem storage.
:::note
You can also use [Redis](http://redis.io/) with Anubis.
:::
This backend is ideal if you are running multiple instances of Anubis in a worker pool (eg: Kubernetes Deployments with a copy of Anubis in each Pod).
| Should I use this backend? | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of Anubis for this service? | 🚫 No |
| Does your service get a lot of traffic? | ✅ Yes |
| Do you want to store data persistently when Anubis restarts? | ✅ Yes |
| Do you run Anubis without mutable filesystem storage? | ✅ Yes |
| Do you have Redis or Valkey installed? | ✅ Yes |
#### Configuration
The `valkey` backend takes the following configuration options:
| Name | Type | Example | Description |
| :---- | :----- | :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | string | `redis://valkey:6379/0` | The URL for the instance of Redis or Valkey that Anubis should store data in. This is in the same format as `REDIS_URL` in many cloud providers. |
Example:
If you have an instance of Valkey running with the hostname `valkey.int.techaro.lol`, then your store configuration could look like this:
```yaml
store:
backend: valkey
parameters:
url: "redis://valkey.int.techaro.lol:6379/0"
```
This would have the Valkey client connect to host `valkey.int.techaro.lol` on port `6379` with database `0` (the default database).
## Risk calculation for downstream services
In case your service needs it for risk calculation reasons, Anubis exposes information about the rules that any requests match using a few headers:

View File

@@ -2,7 +2,7 @@
title: Why does Anubis use Proof-of-Work?
---
Anubis uses a [proof of work](https://en.wikipedia.org/wiki/Proof_of_work) in order to validate that clients are genuine. The reason Anubis does this was inspired by [Hashcash](https://en.wikipedia.org/wiki/Hashcash), a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation. This is also how Bitcoin's consensus algorithm works.
Anubis uses [proof of work](https://en.wikipedia.org/wiki/Proof_of_work) in order to validate that clients are genuine. The reason Anubis does this was inspired by [Hashcash](https://en.wikipedia.org/wiki/Hashcash), a suggestion from the early 2000's about extending the email protocol to avoid spam. The idea is that genuine people sending emails will have to do a small math problem that is expensive to compute, but easy to verify such as hashing a string with a given number of leading zeroes. This will have basically no impact on individuals sending a few emails a week, but the company churning out industrial quantities of advertising will be required to do prohibitively expensive computation. This is also how Bitcoin's consensus algorithm works.
## How Anubis' proof of work scheme works
@@ -21,16 +21,3 @@ const hash = await sha256(`${challenge}${nonce}`);
In order to pass a challenge, the `hash` has to have the right number of leading zeros (the "difficulty"). When a client requests to pass the challenge, they include the nonce they used. The server then only has to do one sha256 operation: the one that confirms that the challenge (generated from request metadata) and the nonce (provided by the client) match the difficulty number of leading zeroes.
Ultimately, this is a hack whose real purpose is to give a "good enough" placeholder solution so that more time can be spent on fingerprinting and identifying headless browsers (EG via how they do font rendering) so that the challenge proof of work page doesn't need to be presented to known legitimate users.
## Challenge format
Anubis generates challenges based on browser metadata, including but not limited to the following:
- The contents of your [`Accept-Language` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Language)
- The IP address of your client
- Your browser's [`User-Agent` string](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent)
- The date of the current week, rooted on Sundays
- Anubis' ed25519 public signing key for [JSON web tokens](https://jwt.io/) (JWTs)
- The challenge difficulty
This is intended to be a random value that is difficult for attackers to forge and guess, but also deterministic enough that it will naturally reset itself.

View File

@@ -21,8 +21,4 @@ If you use a browser extension such as [JShelter](https://jshelter.org/), you wi
## Does Anubis mine Bitcoin?
No. Anubis does not mine Bitcoin.
In order to mine bitcoin, you need to download a copy of the blockchain (so you have the state required to do mining) and also broadcast your mined blocks to the network should you reach a hash with the right number of leading zeroes. You also need to continuously read for newly broadcasted transactions so you can batch them into a block. This requires gigabytes of data to be transferred from the server to the client.
Anubis transfers two digit numbers of kilobytes from the server to the client (which you can independently verify with your browser's Developer Tools feature). This is orders of magnitude below what is required to mine Bitcoin.
No. Anubis does not mine Bitcoin or any other cryptocurrency.

View File

@@ -45,6 +45,11 @@ This page contains a non-exhaustive list with all websites using Anubis.
- https://gitlab.postmarketos.org/
- https://wiki.koha-community.org/
- https://extensions.typo3.org/
- https://ebird.org/
- https://fabulous.systems/
- https://coinhoards.org/
- https://pluralpedia.org/
- https://git.aya.so/
- <details>
<summary>FreeCAD</summary>
- https://forum.freecad.org/
@@ -82,3 +87,10 @@ This page contains a non-exhaustive list with all websites using Anubis.
- https://karla.hds.hebis.de/
- and many more (see https://www.hebis.de/dienste/hebis-discovery-system/)
</details>
- <details>
<summary>Duke University</summary>
- https://repository.duke.edu/
- https://archives.lib.duke.edu/
- https://find.library.duke.edu/
- https://nicholas.duke.edu/
</details>

View File

@@ -11,6 +11,7 @@
## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
bots:
- import: (data)/crawlers/commoncrawl.yaml
# Pathological bots to deny
- # This correlates to data/bots/deny-pathological.yaml in the source tree
# https://github.com/TecharoHQ/anubis/blob/main/data/bots/deny-pathological.yaml
@@ -126,3 +127,8 @@ impressum:
status_codes:
CHALLENGE: 200
DENY: 200
store:
backend: bbolt
parameters:
path: /xe/data/anubis/data.bdb

View File

@@ -15,6 +15,8 @@ spec:
- name: anubis
configMap:
name: anubis-cfg
- name: temporary-data
emptyDir: {}
containers:
- name: anubis-docs
image: ghcr.io/techarohq/anubis/docs:main
@@ -51,6 +53,8 @@ spec:
volumeMounts:
- name: anubis
mountPath: /xe/cfg/anubis
- name: temporary-data
mountPath: /xe/data/anubis
resources:
limits:
cpu: 500m

View File

@@ -4,7 +4,7 @@
"private": true,
"scripts": {
"docusaurus": "docusaurus",
"start": "docusaurus start",
"start": "docusaurus start --host 0.0.0.0",
"build": "docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "echo 'use CI' && exit 1",
@@ -45,4 +45,4 @@
"engines": {
"node": ">=18.0"
}
}
}

Binary file not shown.

After

Width:  |  Height:  |  Size: 29 KiB

BIN
docs/static/img/botstopper/happy.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.5 KiB

BIN
docs/static/img/botstopper/pensive.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.0 KiB

BIN
docs/static/img/botstopper/reject.webp vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.1 KiB

44
go.mod
View File

@@ -10,13 +10,16 @@ require (
github.com/gaissmai/bart v0.20.4
github.com/golang-jwt/jwt/v5 v5.2.2
github.com/google/cel-go v0.25.0
github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus v1.1.0
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.3.2
github.com/joho/godotenv v1.5.1
github.com/nicksnyder/go-i18n/v2 v2.6.0
github.com/playwright-community/playwright-go v0.5200.0
github.com/prometheus/client_golang v1.22.0
github.com/redis/go-redis/v9 v9.11.0
github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a
go.etcd.io/bbolt v1.4.2
golang.org/x/net v0.41.0
golang.org/x/text v0.26.0
google.golang.org/grpc v1.73.0
@@ -31,6 +34,7 @@ require (
cel.dev/expr v0.23.1 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/AlekSi/pointer v1.2.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/BurntSushi/toml v1.5.0 // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.3.1 // indirect
@@ -49,31 +53,45 @@ require (
github.com/cli/browser v1.3.0 // indirect
github.com/cli/go-gh v0.1.0 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.6 // indirect
github.com/cyphar/filepath-securejoin v0.4.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/deckarep/golang-set/v2 v2.8.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/dlclark/regexp2 v1.11.4 // indirect
github.com/docker/docker v28.0.1+incompatible // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c // indirect
github.com/ebitengine/purego v0.8.2 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51 // indirect
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 // indirect
github.com/fatih/color v1.17.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-git/go-git/v5 v5.14.0 // indirect
github.com/go-jose/go-jose/v3 v3.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-yaml v1.12.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/go-github/v70 v70.0.0 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/pprof v0.0.0-20230207041349-798e818bf904 // indirect
github.com/google/rpmpack v0.6.1-0.20250405124433-758cc6896cbc // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/goreleaser/chglog v0.7.0 // indirect
github.com/goreleaser/fileglob v1.3.0 // indirect
github.com/goreleaser/nfpm/v2 v2.42.1 // indirect
@@ -83,33 +101,57 @@ require (
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.5.0 // indirect
github.com/moby/sys/user v0.1.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/natefinch/atomic v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pjbgf/sha1cd v0.3.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/shirou/gopsutil/v4 v4.25.1 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
github.com/spf13/afero v1.14.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/stretchr/testify v1.10.0 // indirect
github.com/suzuki-shunsuke/logrus-error v0.1.4 // indirect
github.com/suzuki-shunsuke/pinact v1.6.0 // indirect
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4 // indirect
github.com/testcontainers/testcontainers-go v0.37.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/ulikunitz/xz v0.5.12 // indirect
github.com/urfave/cli/v2 v2.27.6 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
gitlab.com/digitalxero/go-conventional-commit v1.0.7 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.35.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.opentelemetry.io/otel/trace v1.35.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v3 v3.0.3 // indirect
golang.org/x/crypto v0.39.0 // indirect

91
go.sum
View File

@@ -8,6 +8,8 @@ dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
github.com/AlekSi/pointer v1.2.0 h1:glcy/gc4h8HnG2Z3ZECSzZ1IX1x2JxRVuDzaJwQE0+w=
github.com/AlekSi/pointer v1.2.0/go.mod h1:gZGfd3dpW4vEc/UlyfKKi1roIqcCgwOIvb0tSNSBle0=
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8=
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/BurntSushi/toml v1.5.0 h1:W5quZX/G/csjUnuI8SUYlsHs9M38FC7znL0lIO+DvMg=
github.com/BurntSushi/toml v1.5.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/DataDog/zstd v1.5.5 h1:oWf5W7GtOLgp6bciQYDmhHHjdhYkALu6S/5Ni9ZgSvQ=
@@ -50,6 +52,10 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb h1:m935MPodAbYS46DG4pJSv7WO+VECIWUQ7OJYSoTrMh4=
github.com/blakesmith/ar v0.0.0-20190502131153-809d4375e1fb/go.mod h1:PkYb9DJNAwrSvRx5DYA+gUcOIgTGVMNkfSCbZM8cWpI=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/caarlos0/testfs v0.4.4 h1:3PHvzHi5Lt+g332CiShwS8ogTgS3HjrmzZxCm6JCDr8=
github.com/caarlos0/testfs v0.4.4/go.mod h1:bRN55zgG4XCUVVHZCeU+/Tz1Q6AxEJOEJTliBy+1DMk=
github.com/cavaliergopher/cpio v1.0.1 h1:KQFSeKmZhv0cr+kawA3a0xTQCU4QxXF1vhU7P7av2KM=
@@ -69,6 +75,12 @@ github.com/cli/safeexec v1.0.0/go.mod h1:Z/D4tTN8Vs5gXYHDCbaM1S/anmEDnJb1iW0+EJ5
github.com/cli/shurcooL-graphql v0.0.1/go.mod h1:U7gCSuMZP/Qy7kbqkk5PrqXEeDgtfG5K+W+u8weorps=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/cpuguy83/go-md2man/v2 v2.0.6 h1:XJtiaUW6dEEqVuZiMTn1ldk455QWwEIsMIJlo5vtkx0=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
@@ -81,10 +93,22 @@ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/deckarep/golang-set/v2 v2.8.0 h1:swm0rlPCmdWn9mESxKOjWk8hXSqoxOp+ZlfuyaAdFlQ=
github.com/deckarep/golang-set/v2 v2.8.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/dlclark/regexp2 v1.11.4 h1:rPYF9/LECdNymJufQKmri9gV604RvvABwgOA8un7yAo=
github.com/dlclark/regexp2 v1.11.4/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/docker/docker v28.0.1+incompatible h1:FCHjSRdXhNRFjlHMTv4jUNlIBbTeRjrWfeFuJp7jpo0=
github.com/docker/docker v28.0.1+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c h1:mxWGS0YyquJ/ikZOjSrRjjFIbUqIP9ojyYQ+QZTU3Rg=
github.com/dop251/goja v0.0.0-20250309171923-bcd7cc6bf64c/go.mod h1:MxLav0peU43GgvwVgNbLAj1s/bSGboKkhuULvq/7hx4=
github.com/ebitengine/purego v0.8.2 h1:jPPGWs2sZ1UgOSgD2bClL0MJIqu58nOmIcBuXr62z1I=
github.com/ebitengine/purego v0.8.2/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/elazarl/goproxy v1.7.2 h1:Y2o6urb7Eule09PjlhQRGNsqRfPmYI3KKQLFpCAV3+o=
github.com/elazarl/goproxy v1.7.2/go.mod h1:82vkLNir0ALaW14Rc399OTTjyNREgmdL2cVoIbS6XaE=
github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
@@ -101,6 +125,8 @@ github.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGE
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/fatih/color v1.17.0 h1:GlRw1BRJxkpqUCBKzKOw098ed57fEsKeNjpTe3cSjK4=
github.com/fatih/color v1.17.0/go.mod h1:YZ7TlrGPkiz6ku9fK3TLD/pl3CpsiFyu8N92HLgmosI=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
@@ -119,10 +145,13 @@ github.com/go-git/go-git/v5 v5.14.0 h1:/MD3lCrGjCen5WfEAzKg00MJJffKhC8gzS80ycmCi
github.com/go-git/go-git/v5 v5.14.0/go.mod h1:Z5Xhoia5PcWA3NF8vRLURn9E5FRhSl7dGj9ItW3Wk5k=
github.com/go-jose/go-jose/v3 v3.0.4 h1:Wp5HA7bLQcKnf6YYao/4kpRpVMp/yf6+pJKV8WFSaNY=
github.com/go-jose/go-jose/v3 v3.0.4/go.mod h1:5b+7YgP7ZICgJDBdfjZaIt+H/9L9T/YQrVfLAMboGkQ=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
@@ -142,6 +171,8 @@ github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJA
github.com/goccy/go-yaml v1.9.5/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=
github.com/goccy/go-yaml v1.12.0 h1:/1WHjnMsI1dlIBQutrvSMGZRQufVO3asrHfTwfACoPM=
github.com/goccy/go-yaml v1.12.0/go.mod h1:wKnAMd44+9JAAnGQpWVEgBzGt3YuTaQ4uXoHvE4m7WU=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 h1:f+oWsMOmNPc8JmEHVZIycC7hBoQxHH9pNKQORJNozsQ=
@@ -153,6 +184,7 @@ github.com/google/cel-go v0.25.0/go.mod h1:hjEb6r5SuOSlhCHmFoLzu8HGCERvIsDAbxDAy
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786 h1:rcv+Ippz6RAtvaGgKxc+8FQIpxHgsF+HBzPyYL2cyVU=
github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786/go.mod h1:apVn/GCasLZUVpAJ6oWAuyP7Ne7CEsQbTnc0plM3m+o=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
@@ -196,6 +228,8 @@ github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d h1:RnWZeH8N8KXfbwMTex/KKMYMj0FJRCF6tQubUuQ02GM=
github.com/kjk/lzma v0.0.0-20161016003348-3fd93898850d/go.mod h1:phT/jsRPBAEqjAibu1BurrabCBNTYiVI+zbmyCZJY6Q=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
@@ -213,6 +247,10 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/matryer/is v1.4.0 h1:sosSmIWwkYITGrxZ25ULNDeKiMNzFSr4V/eqBQP0PeE=
github.com/matryer/is v1.4.0/go.mod h1:8I/i5uYgLzgsgEloJE1U6xx5HkBQpAZvepWuujKwMRU=
github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
@@ -232,6 +270,20 @@ github.com/mitchellh/go-ps v1.0.0 h1:i6ampVEEF4wQFF+bkYfwYgY+F/uYJDktmvLPf7qIgjc
github.com/mitchellh/go-ps v1.0.0/go.mod h1:J4lOc8z8yJs6vUwklHw2XEIiT4z4C40KtWVN3nvg8Pg=
github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc=
github.com/moby/sys/sequential v0.5.0/go.mod h1:tH2cOOs5V9MlPiXcQzRC+eEyab644PWKGRYaaV5ZZlo=
github.com/moby/sys/user v0.1.0 h1:WmZ93f5Ux6het5iituh9x2zAG7NFY9Aqi49jjE1PaQg=
github.com/moby/sys/user v0.1.0/go.mod h1:fKJhFOnsCN6xZ5gSfbM6zaHGgDJMrqt9/reuj4T7MmU=
github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/natefinch/atomic v1.0.1 h1:ZPYKxkqQOx3KZ+RsbnP/YsgvxWQPGxjC0oBt2AhwV0A=
@@ -241,6 +293,10 @@ github.com/nicksnyder/go-i18n/v2 v2.6.0 h1:C/m2NNWNiTB6SK4Ao8df5EWm3JETSTIGNXBpM
github.com/nicksnyder/go-i18n/v2 v2.6.0/go.mod h1:88sRqr0C6OPyJn0/KRNaEz1uWorjxIKP7rUUcvycecE=
github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/pjbgf/sha1cd v0.3.2 h1:a9wb0bp1oC2TGwStyn0Umc/IGKQnEgF0vVaZ8QF8eo4=
github.com/pjbgf/sha1cd v0.3.2/go.mod h1:zQWigSxVmsHEZow5qaLtPYxpcKMMQpa09ixqBxuCS6A=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -250,6 +306,8 @@ github.com/playwright-community/playwright-go v0.5200.0/go.mod h1:UnnyQZaqUOO5yw
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
@@ -258,6 +316,8 @@ github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ
github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/redis/go-redis/v9 v9.11.0 h1:E3S08Gl/nJNn5vkxd2i78wZxWAPNZgUNTp8WIJUAiIs=
github.com/redis/go-redis/v9 v9.11.0/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
@@ -269,6 +329,8 @@ github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a h1:iLcLb5Fwwz7g/DLK89F+
github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a/go.mod h1:wozgYq9WEBQBaIJe4YZ0qTSFAMxmcwBhQH0fO0R34Z0=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 h1:n661drycOFuPLCN3Uc8sB6B/s6Z4t2xvBgU1htSHuq8=
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/shirou/gopsutil/v4 v4.25.1 h1:QSWkTc+fu9LTAWfkZwZ6j8MSUk4A2LV7rbH0ZqmLjXs=
github.com/shirou/gopsutil/v4 v4.25.1/go.mod h1:RoUCUpndaJFtT+2zsZzzmhvbfGoDCJ7nFXKJf8GqJbI=
github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
@@ -303,7 +365,13 @@ github.com/suzuki-shunsuke/pinact v1.6.0 h1:2QvSzREOquwLwKXhF9Hj0AInE/Rl63SZz9dK
github.com/suzuki-shunsuke/pinact v1.6.0/go.mod h1:FDUMck0mmL0mcnNZ23Vjh/aOR5cIdZhF1IIpGksT4dQ=
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4 h1:YGHgrVjGTYHY98II6zijXUHP+OyvrzSCvd8m9iUcaK8=
github.com/suzuki-shunsuke/urfave-cli-help-all v0.0.4/go.mod h1:sSi6xaUaHfaqu32ECLeyE7NTMv+ZM5dW0JikhllaalY=
github.com/testcontainers/testcontainers-go v0.37.0 h1:L2Qc0vkTw2EHWQ08djon0D2uw7Z/PtHS/QzZZ5Ra/hg=
github.com/testcontainers/testcontainers-go v0.37.0/go.mod h1:QPzbxZhQ6Bclip9igjLFj6z0hs01bU8lrl2dHQmgFGM=
github.com/thlib/go-timezone-local v0.0.0-20210907160436-ef149e42d28e/go.mod h1:/Tnicc6m/lsJE0irFMA0LfIwTBo4QP7A8IfyIv4zZKI=
github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=
github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
github.com/ulikunitz/xz v0.5.12 h1:37Nm15o69RwBkXM0J6A5OlE67RZTfzUxTj8fB3dfcsc=
github.com/ulikunitz/xz v0.5.12/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/urfave/cli/v2 v2.27.6 h1:VdRdS98FNhKZ8/Az8B7MTyGQmpIr36O1EHybx/LaZ4g=
@@ -314,11 +382,19 @@ github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofm
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
gitlab.com/digitalxero/go-conventional-commit v1.0.7 h1:8/dO6WWG+98PMhlZowt/YjuiKhqhGlOCwlIV8SqqGh8=
gitlab.com/digitalxero/go-conventional-commit v1.0.7/go.mod h1:05Xc2BFsSyC5tKhK0y+P3bs0AwUtNuTp+mTpbCU/DZ0=
go.etcd.io/bbolt v1.4.2 h1:IrUHp260R8c+zYx/Tm8QZr04CX+qWS5PGfPdevhdm1I=
go.etcd.io/bbolt v1.4.2/go.mod h1:Is8rSHO/b4f3XigBC0lL0+4FwAQv3HXEEIgFMuKHceM=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 h1:jq9TW8u3so/bN+JPT166wjOI6/vQPF6Xe7nMNIltagk=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0/go.mod h1:p8pYQP+m5XfbZm9fxtSKAbM6oIllS7s2AfxrChvc7iw=
go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
go.opentelemetry.io/otel/metric v1.35.0 h1:0znxYu2SNyuMSQT4Y9WDWej0VpcsxkuklLa4/siN90M=
@@ -334,6 +410,7 @@ go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
go.yaml.in/yaml/v3 v3.0.3 h1:bXOww4E/J3f66rav3pX3m8w6jDE4knZjGOw8b5Y6iNE=
go.yaml.in/yaml/v3 v3.0.3/go.mod h1:tBHosrYAkRZjRAOREWbDnBXUf08JOwYq++0QNwQiWzI=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
@@ -344,12 +421,16 @@ golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ=
golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.25.0 h1:n7a+ZbQKQA/Ysbyb0/6IbB1H/X41mKgbhfv7AfG/44w=
golang.org/x/mod v0.25.0/go.mod h1:IXM97Txy2VM4PJ3gI61r1YEk/gAj6zAHN3AdZt6S9Ww=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
@@ -360,20 +441,26 @@ golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
golang.org/x/oauth2 v0.28.0 h1:CrgCKl8PPAVtLnU3c+EDw6x11699EWlsDeWNWKdIOkc=
golang.org/x/oauth2 v0.28.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210319071255-635bc2c9138d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -384,6 +471,7 @@ golang.org/x/sys v0.0.0-20220818161305-2296e01440c6/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
@@ -408,6 +496,8 @@ golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo=
@@ -415,6 +505,7 @@ golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg
golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=

View File

@@ -1,6 +1,7 @@
package ogtags
import (
"context"
"errors"
"log/slog"
"net/url"
@@ -8,7 +9,7 @@ import (
)
// GetOGTags is the main function that retrieves Open Graph tags for a URL
func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]string, error) {
func (c *OGTagCache) GetOGTags(ctx context.Context, url *url.URL, originalHost string) (map[string]string, error) {
if url == nil {
return nil, errors.New("nil URL provided, cannot fetch OG tags")
}
@@ -21,12 +22,12 @@ func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]st
cacheKey := c.generateCacheKey(target, originalHost)
// Check cache first
if cachedTags := c.checkCache(cacheKey); cachedTags != nil {
if cachedTags := c.checkCache(ctx, cacheKey); cachedTags != nil {
return cachedTags, nil
}
// Fetch HTML content, passing the original host
doc, err := c.fetchHTMLDocumentWithCache(target, originalHost, cacheKey)
doc, err := c.fetchHTMLDocumentWithCache(ctx, target, originalHost, cacheKey)
if errors.Is(err, syscall.ECONNREFUSED) {
slog.Debug("Connection refused, returning empty tags")
return nil, nil
@@ -42,7 +43,7 @@ func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]st
ogTags := c.extractOGTags(doc)
// Store in cache
c.cache.Set(cacheKey, ogTags, c.ogTimeToLive)
c.cache.Set(ctx, cacheKey, ogTags, c.ogTimeToLive)
return ogTags, nil
}
@@ -59,8 +60,8 @@ func (c *OGTagCache) generateCacheKey(target string, originalHost string) string
}
// checkCache checks if we have the tags cached and returns them if so
func (c *OGTagCache) checkCache(cacheKey string) map[string]string {
if cachedTags, ok := c.cache.Get(cacheKey); ok {
func (c *OGTagCache) checkCache(ctx context.Context, cacheKey string) map[string]string {
if cachedTags, err := c.cache.Get(ctx, cacheKey); err == nil {
slog.Debug("cache hit", "tags", cachedTags)
return cachedTags
}

View File

@@ -9,6 +9,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestCacheReturnsDefault(t *testing.T) {
@@ -21,14 +22,14 @@ func TestCacheReturnsDefault(t *testing.T) {
TimeToLive: time.Minute,
ConsiderHost: false,
Override: want,
})
}, memory.New(t.Context()))
u, err := url.Parse("https://anubis.techaro.lol")
if err != nil {
t.Fatal(err)
}
result, err := cache.GetOGTags(u, "anubis.techaro.lol")
result, err := cache.GetOGTags(t.Context(), u, "anubis.techaro.lol")
if err != nil {
t.Fatal(err)
}
@@ -49,7 +50,7 @@ func TestCheckCache(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Set up test data
urlStr := "http://example.com/page"
@@ -60,16 +61,16 @@ func TestCheckCache(t *testing.T) {
cacheKey := cache.generateCacheKey(urlStr, "example.com")
// Test cache miss
tags := cache.checkCache(cacheKey)
tags := cache.checkCache(t.Context(), cacheKey)
if tags != nil {
t.Errorf("expected nil tags on cache miss, got %v", tags)
}
// Manually add to cache
cache.cache.Set(cacheKey, expectedTags, time.Minute)
cache.cache.Set(t.Context(), cacheKey, expectedTags, time.Minute)
// Test cache hit
tags = cache.checkCache(cacheKey)
tags = cache.checkCache(t.Context(), cacheKey)
if tags == nil {
t.Fatal("expected non-nil tags on cache hit, got nil")
}
@@ -112,7 +113,7 @@ func TestGetOGTags(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Parse the test server URL
parsedURL, err := url.Parse(ts.URL)
@@ -122,7 +123,7 @@ func TestGetOGTags(t *testing.T) {
// Test fetching OG tags from the test server
// Pass the host from the parsed test server URL
ogTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
ogTags, err := cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags: %v", err)
}
@@ -142,14 +143,14 @@ func TestGetOGTags(t *testing.T) {
// Test fetching OG tags from the cache
// Pass the host from the parsed test server URL
ogTags, err = cache.GetOGTags(parsedURL, parsedURL.Host)
ogTags, err = cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}
// Test fetching OG tags from the cache (3rd time)
// Pass the host from the parsed test server URL
newOgTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
newOgTags, err := cache.GetOGTags(t.Context(), parsedURL, parsedURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}
@@ -263,10 +264,10 @@ func TestGetOGTagsWithHostConsideration(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: tc.ogCacheConsiderHost,
})
}, memory.New(t.Context()))
for i, req := range tc.requests {
ogTags, err := cache.GetOGTags(parsedURL, req.host)
ogTags, err := cache.GetOGTags(t.Context(), parsedURL, req.host)
if err != nil {
t.Errorf("Request %d (host: %s): unexpected error: %v", i+1, req.host, err)
continue // Skip further checks for this request if error occurred

View File

@@ -20,8 +20,8 @@ var (
// fetchHTMLDocumentWithCache fetches the HTML document from the given URL string,
// preserving the original host header.
func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost string, cacheKey string) (*html.Node, error) {
req, err := http.NewRequestWithContext(context.Background(), "GET", urlStr, nil)
func (c *OGTagCache) fetchHTMLDocumentWithCache(ctx context.Context, urlStr string, originalHost string, cacheKey string) (*html.Node, error) {
req, err := http.NewRequestWithContext(ctx, "GET", urlStr, nil)
if err != nil {
return nil, fmt.Errorf("failed to create http request: %w", err)
}
@@ -41,7 +41,7 @@ func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost stri
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
slog.Debug("og: request timed out", "url", urlStr)
c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
c.cache.Set(ctx, cacheKey, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
}
return nil, fmt.Errorf("http get failed: %w", err)
}
@@ -56,7 +56,7 @@ func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost stri
if resp.StatusCode != http.StatusOK {
slog.Debug("og: received non-OK status code", "url", urlStr, "status", resp.StatusCode)
c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
c.cache.Set(ctx, cacheKey, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
return nil, fmt.Errorf("%w: page not found", ErrOgHandled)
}

View File

@@ -1,6 +1,7 @@
package ogtags
import (
"context"
"fmt"
"io"
"net/http"
@@ -11,6 +12,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -85,8 +87,8 @@ func TestFetchHTMLDocument(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
doc, err := cache.fetchHTMLDocument(ts.URL, "anything")
}, memory.New(t.Context()))
doc, err := cache.fetchHTMLDocument(t.Context(), ts.URL, "anything")
if tt.expectError {
if err == nil {
@@ -116,9 +118,9 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
doc, err := cache.fetchHTMLDocument("http://invalid.url.that.doesnt.exist.example", "anything")
doc, err := cache.fetchHTMLDocument(t.Context(), "http://invalid.url.that.doesnt.exist.example", "anything")
if err == nil {
t.Error("expected error for invalid URL, got nil")
@@ -130,7 +132,7 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
}
// fetchHTMLDocument allows you to call fetchHTMLDocumentWithCache without a duplicate generateCacheKey call
func (c *OGTagCache) fetchHTMLDocument(urlStr string, originalHost string) (*html.Node, error) {
func (c *OGTagCache) fetchHTMLDocument(ctx context.Context, urlStr string, originalHost string) (*html.Node, error) {
cacheKey := c.generateCacheKey(urlStr, originalHost)
return c.fetchHTMLDocumentWithCache(urlStr, originalHost, cacheKey)
return c.fetchHTMLDocumentWithCache(ctx, urlStr, originalHost, cacheKey)
}

View File

@@ -8,6 +8,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestIntegrationGetOGTags(t *testing.T) {
@@ -110,7 +111,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Create URL for test
testURL, _ := url.Parse(ts.URL)
@@ -119,7 +120,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
// Get OG tags
// Pass the host from the test URL
ogTags, err := cache.GetOGTags(testURL, testURL.Host)
ogTags, err := cache.GetOGTags(t.Context(), testURL, testURL.Host)
// Check error expectation
if tc.expectError {
@@ -147,7 +148,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
// Test cache retrieval
// Pass the host from the test URL
cachedOGTags, err := cache.GetOGTags(testURL, testURL.Host)
cachedOGTags, err := cache.GetOGTags(t.Context(), testURL, testURL.Host)
if err != nil {
t.Fatalf("failed to get OG tags from cache: %v", err)
}

View File

@@ -7,6 +7,7 @@ import (
"testing"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -30,7 +31,7 @@ func BenchmarkGetTarget(b *testing.B) {
for _, tt := range tests {
b.Run(tt.name, func(b *testing.B) {
cache := NewOGTagCache(tt.target, config.OpenGraph{})
cache := NewOGTagCache(tt.target, config.OpenGraph{}, memory.New(b.Context()))
urls := make([]*url.URL, len(tt.paths))
for i, path := range tt.paths {
u, _ := url.Parse(path)
@@ -66,7 +67,7 @@ func BenchmarkExtractOGTags(b *testing.B) {
</head><body><div><p>Content</p></div></body></html>`,
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(b.Context()))
docs := make([]*html.Node, len(htmlSamples))
for i, sample := range htmlSamples {
@@ -84,7 +85,7 @@ func BenchmarkExtractOGTags(b *testing.B) {
// Memory usage test
func TestMemoryUsage(t *testing.T) {
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(t.Context()))
// Force GC and wait for it to complete
runtime.GC()

View File

@@ -9,8 +9,8 @@ import (
"strings"
"time"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
)
const (
@@ -22,7 +22,7 @@ const (
)
type OGTagCache struct {
cache *decaymap.Impl[string, map[string]string]
cache store.JSON[map[string]string]
targetURL *url.URL
client *http.Client
@@ -36,7 +36,7 @@ type OGTagCache struct {
ogOverride map[string]string
}
func NewOGTagCache(target string, conf config.OpenGraph) *OGTagCache {
func NewOGTagCache(target string, conf config.OpenGraph, backend store.Interface) *OGTagCache {
// Predefined approved tags and prefixes
defaultApprovedTags := []string{"description", "keywords", "author"}
defaultApprovedPrefixes := []string{"og:", "twitter:", "fediverse:"}
@@ -77,7 +77,10 @@ func NewOGTagCache(target string, conf config.OpenGraph) *OGTagCache {
}
return &OGTagCache{
cache: decaymap.New[string, map[string]string](),
cache: store.JSON[map[string]string]{
Underlying: backend,
Prefix: "ogtags:",
},
targetURL: parsedTargetURL,
ogPassthrough: conf.Enabled,
ogTimeToLive: conf.TimeToLive,
@@ -124,9 +127,3 @@ func (c *OGTagCache) getTarget(u *url.URL) string {
return sb.String()
}
func (c *OGTagCache) Cleanup() {
if c.cache != nil {
c.cache.Cleanup()
}
}

View File

@@ -1,12 +1,14 @@
package ogtags
import (
"context"
"net/url"
"strings"
"testing"
"unicode/utf8"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -46,7 +48,7 @@ func FuzzGetTarget(f *testing.F) {
}
// Create cache - should not panic
cache := NewOGTagCache(target, config.OpenGraph{})
cache := NewOGTagCache(target, config.OpenGraph{}, memory.New(context.Background()))
// Create URL
u := &url.URL{
@@ -130,7 +132,7 @@ func FuzzExtractOGTags(f *testing.F) {
return
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(context.Background()))
// Should not panic
tags := cache.extractOGTags(doc)
@@ -186,7 +188,7 @@ func FuzzGetTargetRoundTrip(f *testing.F) {
t.Skip()
}
cache := NewOGTagCache(target, config.OpenGraph{})
cache := NewOGTagCache(target, config.OpenGraph{}, memory.New(context.Background()))
u := &url.URL{Path: path, RawQuery: query}
result := cache.getTarget(u)
@@ -243,7 +245,7 @@ func FuzzExtractMetaTagInfo(f *testing.F) {
},
}
cache := NewOGTagCache("http://example.com", config.OpenGraph{})
cache := NewOGTagCache("http://example.com", config.OpenGraph{}, memory.New(context.Background()))
// Should not panic
property, content := cache.extractMetaTagInfo(node)
@@ -296,7 +298,7 @@ func BenchmarkFuzzedGetTarget(b *testing.B) {
for _, input := range inputs {
b.Run(input.name, func(b *testing.B) {
cache := NewOGTagCache(input.target, config.OpenGraph{})
cache := NewOGTagCache(input.target, config.OpenGraph{}, memory.New(context.Background()))
u := &url.URL{Path: input.path, RawQuery: input.query}
b.ResetTimer()

View File

@@ -15,6 +15,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestNewOGTagCache(t *testing.T) {
@@ -44,7 +45,7 @@ func TestNewOGTagCache(t *testing.T) {
Enabled: tt.ogPassthrough,
TimeToLive: tt.ogTimeToLive,
ConsiderHost: false,
})
}, memory.New(t.Context()))
if cache == nil {
t.Fatal("expected non-nil cache, got nil")
@@ -84,7 +85,7 @@ func TestNewOGTagCache_UnixSocket(t *testing.T) {
Enabled: true,
TimeToLive: 5 * time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
if cache == nil {
t.Fatal("expected non-nil cache, got nil")
@@ -169,7 +170,7 @@ func TestGetTarget(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
u := &url.URL{
Path: tt.path,
@@ -242,14 +243,14 @@ func TestIntegrationGetOGTags_UnixSocket(t *testing.T) {
Enabled: true,
TimeToLive: time.Minute,
ConsiderHost: false,
})
}, memory.New(t.Context()))
// Create a dummy URL for the request (path and query matter)
testReqURL, _ := url.Parse("/some/page?query=1")
// Get OG tags
// Pass an empty string for host, as it's irrelevant for unix sockets
ogTags, err := cache.GetOGTags(testReqURL, "")
ogTags, err := cache.GetOGTags(t.Context(), testReqURL, "")
if err != nil {
t.Fatalf("GetOGTags failed for unix socket: %v", err)
@@ -265,7 +266,7 @@ func TestIntegrationGetOGTags_UnixSocket(t *testing.T) {
// Test cache retrieval (should hit cache)
// Pass an empty string for host
cachedTags, err := cache.GetOGTags(testReqURL, "")
cachedTags, err := cache.GetOGTags(t.Context(), testReqURL, "")
if err != nil {
t.Fatalf("GetOGTags (cache hit) failed for unix socket: %v", err)
}

View File

@@ -7,6 +7,7 @@ import (
"time"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/memory"
"golang.org/x/net/html"
)
@@ -17,7 +18,7 @@ func TestExtractOGTags(t *testing.T) {
Enabled: false,
ConsiderHost: false,
TimeToLive: time.Minute,
})
}, memory.New(t.Context()))
// Manually set approved tags/prefixes based on the user request for clarity
testCache.approvedTags = []string{"description"}
testCache.approvedPrefixes = []string{"og:"}
@@ -198,7 +199,7 @@ func TestExtractMetaTagInfo(t *testing.T) {
Enabled: false,
ConsiderHost: false,
TimeToLive: time.Minute,
})
}, memory.New(t.Context()))
testCache.approvedTags = []string{"description"}
testCache.approvedPrefixes = []string{"og:"}

22
internal/unbreakdocker.go Normal file
View File

@@ -0,0 +1,22 @@
package internal
import (
"os"
"os/exec"
)
func UnbreakDocker() {
// XXX(Xe): This is bad code. Do not do this.
//
// I have to do this because I'm running from inside the context of a dev
// container. This dev container runs in a different docker network than
// the valkey test container runs in. In order to let my dev container
// connect to the test container, they need to share a network in common.
// The easiest network to use for this is the default "bridge" network.
//
// This is a horrifying monstrosity, but the part that scares me the most
// is the fact that it works.
if hostname, err := os.Hostname(); err == nil {
exec.Command("docker", "network", "connect", "bridge", hostname).Run()
}
}

View File

@@ -1,8 +1,9 @@
package lib
import (
"context"
"crypto/ed25519"
"crypto/sha256"
"crypto/rand"
"encoding/json"
"errors"
"fmt"
@@ -16,6 +17,7 @@ import (
"github.com/golang-jwt/jwt/v5"
"github.com/google/cel-go/common/types"
"github.com/google/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
@@ -30,6 +32,7 @@ import (
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/checker"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
// challenge implementations
_ "github.com/TecharoHQ/anubis/lib/challenge/metarefresh"
@@ -67,11 +70,11 @@ type Server struct {
next http.Handler
mux *http.ServeMux
policy *policy.ParsedConfig
DNSBLCache *decaymap.Impl[string, dnsbl.DroneBLResponse]
OGTags *ogtags.OGTagCache
ed25519Priv ed25519.PrivateKey
hs512Secret []byte
opts Options
store store.Interface
}
func (s *Server) getTokenKeyfunc() jwt.Keyfunc {
@@ -87,23 +90,51 @@ func (s *Server) getTokenKeyfunc() jwt.Keyfunc {
}
}
func (s *Server) challengeFor(r *http.Request, difficulty int) string {
var fp [32]byte
if len(s.hs512Secret) == 0 {
fp = sha256.Sum256(s.ed25519Priv.Public().(ed25519.PublicKey)[:])
} else {
fp = sha256.Sum256(s.hs512Secret)
func (s *Server) challengeFor(r *http.Request) (*challenge.Challenge, error) {
ckies := r.CookiesNamed(anubis.TestCookieName)
if len(ckies) == 0 {
return s.issueChallenge(r.Context(), r)
}
challengeData := fmt.Sprintf(
"X-Real-IP=%s,User-Agent=%s,WeekTime=%s,Fingerprint=%x,Difficulty=%d",
r.Header.Get("X-Real-Ip"),
r.UserAgent(),
time.Now().UTC().Round(24*7*time.Hour).Format(time.RFC3339),
fp,
difficulty,
)
return internal.FastHash(challengeData)
j := store.JSON[challenge.Challenge]{Underlying: s.store}
ckie := ckies[0]
chall, err := j.Get(r.Context(), "challenge:"+ckie.Value)
if err != nil {
return nil, err
}
return &chall, nil
}
func (s *Server) issueChallenge(ctx context.Context, r *http.Request) (*challenge.Challenge, error) {
id, err := uuid.NewV7()
if err != nil {
return nil, err
}
var randomData = make([]byte, 256)
if _, err := rand.Read(randomData); err != nil {
return nil, err
}
chall := challenge.Challenge{
ID: id.String(),
RandomData: fmt.Sprintf("%x", randomData),
IssuedAt: time.Now(),
Metadata: map[string]string{
"User-Agent": r.Header.Get("User-Agent"),
"X-Real-Ip": r.Header.Get("X-Real-Ip"),
},
}
j := store.JSON[challenge.Challenge]{Underlying: s.store}
if err := j.Set(ctx, "challenge:"+id.String(), chall, 30*time.Minute); err != nil {
return nil, err
}
return &chall, err
}
func (s *Server) maybeReverseProxyHttpStatusOnly(w http.ResponseWriter, r *http.Request) {
@@ -247,15 +278,16 @@ func (s *Server) checkRules(w http.ResponseWriter, r *http.Request, cr policy.Ch
}
func (s *Server) handleDNSBL(w http.ResponseWriter, r *http.Request, ip string, lg *slog.Logger) bool {
db := &store.JSON[dnsbl.DroneBLResponse]{Underlying: s.store, Prefix: "dronebl:"}
if s.policy.DNSBL && ip != "" {
resp, ok := s.DNSBLCache.Get(ip)
if !ok {
resp, err := db.Get(r.Context(), ip)
if err != nil {
lg.Debug("looking up ip in dnsbl")
resp, err := dnsbl.Lookup(ip)
if err != nil {
lg.Error("can't look up ip in dnsbl", "err", err)
}
s.DNSBLCache.Set(ip, resp, 24*time.Hour)
db.Set(r.Context(), ip, resp, 24*time.Hour)
droneBLHits.WithLabelValues(resp.String()).Inc()
}
@@ -309,15 +341,30 @@ func (s *Server) MakeChallenge(w http.ResponseWriter, r *http.Request) {
return
}
lg = lg.With("check_result", cr)
chal := s.challengeFor(r, rule.Challenge.Difficulty)
s.SetCookie(w, CookieOpts{Host: r.Host, Name: anubis.TestCookieName, Value: chal})
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("failed to fetch or issue challenge", "err", err)
w.WriteHeader(http.StatusInternalServerError)
err := encoder.Encode(struct {
Error string `json:"error"`
}{
Error: fmt.Sprintf("%s \"makeChallenge\"", localizer.T("internal_server_error")),
})
if err != nil {
lg.Error("failed to encode error response", "err", err)
w.WriteHeader(http.StatusInternalServerError)
}
return
}
s.SetCookie(w, CookieOpts{Host: r.Host, Name: anubis.TestCookieName, Value: chall.ID})
err = encoder.Encode(struct {
Rules *config.ChallengeRules `json:"rules"`
Challenge string `json:"challenge"`
}{
Challenge: chal,
Challenge: chall.RandomData,
Rules: rule.Challenge,
})
if err != nil {
@@ -325,7 +372,7 @@ func (s *Server) MakeChallenge(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
return
}
lg.Debug("made challenge", "challenge", chal, "rules", rule.Challenge, "cr", cr)
lg.Debug("made challenge", "challenge", chall, "rules", rule.Challenge, "cr", cr)
challengesIssued.WithLabelValues("api").Inc()
}
@@ -384,9 +431,20 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
return
}
challengeStr := s.challengeFor(r, rule.Challenge.Difficulty)
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("check failed", "err", err)
s.respondWithError(w, r, fmt.Sprintf("%s: %s", localizer.T("internal_server_error"), rule.Challenge.Algorithm))
return
}
if err := impl.Validate(r, lg, rule, challengeStr); err != nil {
in := &challenge.ValidateInput{
Challenge: chall,
Rule: rule,
Store: s.store,
}
if err := impl.Validate(r, lg, in); err != nil {
failedValidations.WithLabelValues(rule.Challenge.Algorithm).Inc()
var cerr *challenge.Error
s.ClearCookie(w, CookieOpts{Path: cookiePath, Host: r.Host})
@@ -405,7 +463,7 @@ func (s *Server) PassChallenge(w http.ResponseWriter, r *http.Request) {
// generate JWT cookie
tokenString, err := s.signJWT(jwt.MapClaims{
"challenge": challengeStr,
"challenge": chall.ID,
"method": rule.Challenge.Algorithm,
"policyRule": rule.Hash(),
"action": string(cr.Rule),
@@ -493,8 +551,3 @@ func (s *Server) check(r *http.Request) (policy.CheckResult, *policy.Bot, error)
Rules: &checker.List{},
}, nil
}
func (s *Server) CleanupDecayMap() {
s.DNSBLCache.Cleanup()
s.OGTags.Cleanup()
}

View File

@@ -1,60 +1,11 @@
package challenge
import (
"log/slog"
"net/http"
"sort"
"sync"
import "time"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/a-h/templ"
)
var (
registry map[string]Impl = map[string]Impl{}
regLock sync.RWMutex
)
func Register(name string, impl Impl) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Impl, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}
type IssueInput struct {
Impressum *config.Impressum
Rule *policy.Bot
Challenge string
OGTags map[string]string
}
type Impl interface {
// Setup registers any additional routes with the Impl for assets or API routes.
Setup(mux *http.ServeMux)
// Issue a new challenge to the user, called by the Anubis.
Issue(r *http.Request, lg *slog.Logger, in *IssueInput) (templ.Component, error)
// Validate a challenge, making sure that it passes muster.
Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, challenge string) error
// Challenge is the metadata about a single challenge issuance.
type Challenge struct {
ID string `json:"id"` // UUID identifying the challenge
RandomData string `json:"randomData"` // The random data the client processes
IssuedAt time.Time `json:"issuedAt"` // When the challenge was issued
Metadata map[string]string `json:"metadata"` // Challenge metadata such as IP address and user agent
}

View File

@@ -0,0 +1,23 @@
package challengetest
import (
"testing"
"time"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/google/uuid"
)
func New(t *testing.T) *challenge.Challenge {
t.Helper()
id := uuid.Must(uuid.NewV7())
randomData := internal.SHA256sum(time.Now().String())
return &challenge.Challenge{
ID: id.String(),
RandomData: randomData,
IssuedAt: time.Now(),
}
}

View File

@@ -0,0 +1,7 @@
package challengetest
import "testing"
func TestNew(t *testing.T) {
_ = New(t)
}

View File

@@ -0,0 +1,68 @@
package challenge
import (
"log/slog"
"net/http"
"sort"
"sync"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/a-h/templ"
)
var (
registry map[string]Impl = map[string]Impl{}
regLock sync.RWMutex
)
func Register(name string, impl Impl) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Impl, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}
type IssueInput struct {
Impressum *config.Impressum
Rule *policy.Bot
Challenge *Challenge
OGTags map[string]string
Store store.Interface
}
type ValidateInput struct {
Rule *policy.Bot
Challenge *Challenge
Store store.Interface
}
type Impl interface {
// Setup registers any additional routes with the Impl for assets or API routes.
Setup(mux *http.ServeMux)
// Issue a new challenge to the user, called by the Anubis.
Issue(r *http.Request, lg *slog.Logger, in *IssueInput) (templ.Component, error)
// Validate a challenge, making sure that it passes muster.
Validate(r *http.Request, lg *slog.Logger, in *ValidateInput) error
}

View File

@@ -9,7 +9,6 @@ import (
"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/web"
"github.com/a-h/templ"
)
@@ -32,11 +31,11 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *challenge.IssueInput)
q := u.Query()
q.Set("redir", r.URL.String())
q.Set("challenge", in.Challenge)
q.Set("challenge", in.Challenge.RandomData)
u.RawQuery = q.Encode()
loc := localization.GetLocalizer(r)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), page(in.Challenge, u.String(), in.Rule.Challenge.Difficulty, loc), in.Impressum, in.Challenge, in.Rule.Challenge, in.OGTags, loc)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), page(u.String(), in.Rule.Challenge.Difficulty, loc), in.Impressum, in.Challenge.RandomData, in.Rule.Challenge, in.OGTags, loc)
if err != nil {
return nil, fmt.Errorf("can't render page: %w", err)
@@ -45,11 +44,11 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *challenge.IssueInput)
return component, nil
}
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, wantChallenge string) error {
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, in *challenge.ValidateInput) error {
gotChallenge := r.FormValue("challenge")
if subtle.ConstantTimeCompare([]byte(wantChallenge), []byte(gotChallenge)) != 1 {
return challenge.NewError("validate", "invalid response", fmt.Errorf("%w: wanted response %s but got %s", challenge.ErrFailed, wantChallenge, gotChallenge))
if subtle.ConstantTimeCompare([]byte(in.Challenge.RandomData), []byte(gotChallenge)) != 1 {
return challenge.NewError("validate", "invalid response", fmt.Errorf("%w: wanted response %s but got %s", challenge.ErrFailed, in.Challenge.RandomData, gotChallenge))
}
return nil

View File

@@ -7,7 +7,7 @@ import (
"github.com/TecharoHQ/anubis/lib/localization"
)
templ page(challenge, redir string, difficulty int, loc *localization.SimpleLocalizer) {
templ page(redir string, difficulty int, loc *localization.SimpleLocalizer) {
<div class="centered-div">
<img id="image" style="width:100%;max-width:256px;" src={ anubis.BasePrefix + "/.within.website/x/cmd/anubis/static/img/pensive.webp?cacheBuster=" + anubis.Version }/>
<img style="display:none;" style="width:100%;max-width:256px;" src={ anubis.BasePrefix + "/.within.website/x/cmd/anubis/static/img/happy.webp?cacheBuster=" + anubis.Version }/>

View File

@@ -1,6 +1,6 @@
// Code generated by templ - DO NOT EDIT.
// templ: version: v0.3.898
// templ: version: v0.3.906
package metarefresh
//lint:file-ignore SA4006 This context is only used if a nested component is present.
@@ -15,7 +15,7 @@ import (
"github.com/TecharoHQ/anubis/lib/localization"
)
func page(challenge, redir string, difficulty int, loc *localization.SimpleLocalizer) templ.Component {
func page(redir string, difficulty int, loc *localization.SimpleLocalizer) templ.Component {
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {

View File

@@ -11,7 +11,6 @@ import (
"github.com/TecharoHQ/anubis/internal"
chall "github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
"github.com/TecharoHQ/anubis/lib/policy"
"github.com/TecharoHQ/anubis/web"
"github.com/a-h/templ"
)
@@ -31,7 +30,7 @@ func (i *Impl) Setup(mux *http.ServeMux) {
func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *chall.IssueInput) (templ.Component, error) {
loc := localization.GetLocalizer(r)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), web.Index(loc), in.Impressum, in.Challenge, in.Rule.Challenge, in.OGTags, loc)
component, err := web.BaseWithChallengeAndOGTags(loc.T("making_sure_not_bot"), web.Index(loc), in.Impressum, in.Challenge.RandomData, in.Rule.Challenge, in.OGTags, loc)
if err != nil {
return nil, fmt.Errorf("can't render page: %w", err)
}
@@ -39,7 +38,10 @@ func (i *Impl) Issue(r *http.Request, lg *slog.Logger, in *chall.IssueInput) (te
return component, nil
}
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, rule *policy.Bot, challenge string) error {
func (i *Impl) Validate(r *http.Request, lg *slog.Logger, in *chall.ValidateInput) error {
rule := in.Rule
challenge := in.Challenge.RandomData
nonceStr := r.FormValue("nonce")
if nonceStr == "" {
return chall.NewError("validate", "invalid response", fmt.Errorf("%w nonce", chall.ErrMissingField))

View File

@@ -124,16 +124,25 @@ func TestBasic(t *testing.T) {
t.Run(cs.name, func(t *testing.T) {
lg := slog.With()
i.Setup(http.NewServeMux())
inp := &challenge.IssueInput{
Rule: bot,
Challenge: cs.challengeStr,
Rule: bot,
Challenge: &challenge.Challenge{
RandomData: cs.challengeStr,
},
}
if _, err := i.Issue(cs.req, lg, inp); err != nil {
t.Errorf("can't issue challenge: %v", err)
}
if err := i.Validate(cs.req, lg, bot, cs.challengeStr); !errors.Is(err, cs.err) {
if err := i.Validate(cs.req, lg, &challenge.ValidateInput{
Rule: bot,
Challenge: &challenge.Challenge{
RandomData: cs.challengeStr,
},
}); !errors.Is(err, cs.err) {
t.Errorf("got wrong error from Validate, got %v but wanted %v", err, cs.err)
}
})

View File

@@ -15,9 +15,7 @@ import (
"github.com/TecharoHQ/anubis"
"github.com/TecharoHQ/anubis/data"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/internal/dnsbl"
"github.com/TecharoHQ/anubis/internal/ogtags"
"github.com/TecharoHQ/anubis/lib/challenge"
"github.com/TecharoHQ/anubis/lib/localization"
@@ -108,8 +106,8 @@ func New(opts Options) (*Server, error) {
hs512Secret: opts.HS512Secret,
policy: opts.Policy,
opts: opts,
DNSBLCache: decaymap.New[string, dnsbl.DroneBLResponse](),
OGTags: ogtags.NewOGTagCache(opts.Target, opts.Policy.OpenGraph),
OGTags: ogtags.NewOGTagCache(opts.Target, opts.Policy.OpenGraph, opts.Policy.Store),
store: opts.Policy.Store,
}
mux := http.NewServeMux()

View File

@@ -128,19 +128,24 @@ func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, rule *polic
}
challengesIssued.WithLabelValues("embedded").Add(1)
challengeStr := s.challengeFor(r, rule.Challenge.Difficulty)
chall, err := s.challengeFor(r)
if err != nil {
lg.Error("can't get challenge", "err", "err")
s.respondWithError(w, r, fmt.Sprintf("%s: %s", localizer.T("internal_server_error"), rule.Challenge.Algorithm))
return
}
var ogTags map[string]string = nil
if s.opts.OpenGraph.Enabled {
var err error
ogTags, err = s.OGTags.GetOGTags(r.URL, r.Host)
ogTags, err = s.OGTags.GetOGTags(r.Context(), r.URL, r.Host)
if err != nil {
lg.Error("failed to get OG tags", "err", err)
}
}
s.SetCookie(w, CookieOpts{
Value: challengeStr,
Value: chall.ID,
Host: r.Host,
Path: "/",
Name: anubis.TestCookieName,
@@ -157,8 +162,9 @@ func (s *Server) RenderIndex(w http.ResponseWriter, r *http.Request, rule *polic
in := &challenge.IssueInput{
Impressum: s.policy.Impressum,
Rule: rule,
Challenge: challengeStr,
Challenge: chall,
OGTags: ogTags,
Store: s.store,
}
component, err := impl.Issue(r, lg, in)

View File

@@ -0,0 +1,63 @@
{
"loading": "Ladevorgang...",
"why_am_i_seeing": "Warum sehe ich diese Seite?",
"protected_by": "Geschützt durch",
"made_with": "Mit ❤️ gemacht in 🇨🇦",
"mascot_design": "Maskottchen erstellt von",
"ai_companies_explanation": "Diese Seite wird angezeigt, da der Betreiber der Webseite Anubis eingerichtet hat, um sie vor aggressiven KI-Website-Scrapern zu schützen. Diese können Ausfälle der Webseite verursachen, wodurch die Webseite für jeden nicht erreichbar ist.",
"anubis_compromise": "Anubis ist eine Art Kompromiss. Es verwendet die sogenannte Proof-of-Work Methode nach Hashcash, ein Mechanismus, der ursprünglich zur E-Mail-Spam-Bekämpfung entwickelt wurde. Die Idee dahinter ist, dass ein einziger User nur eine kleine Verzögerung hat, auf die Webseite zu gelangen; bei Scrapern kann das allerdings große Auswirkungen haben.",
"hack_purpose": "Man könnte dies als eine Lösung bezeichnen, die gut genug ist, einem etwas Zeit zu verschaffen für Fingerprinting und dem Identifizieren von Headless Browsern, sodass im besten Fall normale User diese Seite garnicht erst zu sehen bekommen.",
"jshelter_note": "Anubis benötigt moderne JavaScript-Features, welche von Plugins wie zB JShelter deaktiviert werden. Bitte deaktiviere also JShelter oder ähnliche Plugins für diese Domain.",
"version_info": "Diese Webseite läuft mit Anubis version",
"try_again": "Nochmal probieren",
"go_home": "Zur Hauptseite",
"contact_webmaster": "oder wenn es sich hier um einen Fehler handelt, kontaktiere bitte den Administrator der Webseite unter",
"connection_security": "Bitte warte einen Moment während wir sicherstellen, dass eine sichere Verbindung verwendet wird.",
"javascript_required": "Es muss leider JavaScript aktiviert werden, um den Check durchführen zu können. Dies ist leider notwendig weil Firmen im KI-Sektor die sozialen Verhältnisse geändert haben, wie Website-Hosting funktioniert. Eine Lösung ohne JavaScript ist in Entwicklung.",
"benchmark_requires_js": "Das Benchmark-Tool benötigt das Aktivieren von JavaScript.",
"difficulty": "Schwierigkeit:",
"algorithm": "Algorithmus:",
"compare": "Vergleich:",
"time": "Zeit",
"iters": "Iterationen",
"time_a": "Zeit A",
"iters_a": "Iterationen A",
"time_b": "Zeit B",
"iters_b": "Iterationen B",
"static_check_endpoint": "Dies ist nur ein Check-Endpunkt, der von beispielsweise einem Reverse-Proxy geprüft werden kann.",
"authorization_required": "Zugriffserlaubnis benötigt",
"cookies_disabled": "Cookies sind in Ihrem Browser deaktiviert. Anubis benötigt Cookies um sicherzustellen, dass es sich hierbei um einen validen Zugriff handelt. Bitte aktiviere Cookies für diese Domain.",
"access_denied": "Zugriff verweigert: Fehlercode",
"dronebl_entry": "Eintrag in DroneBL",
"see_dronebl_lookup": "anzeigen",
"internal_server_error": "Interner Server Error: Misskonfiguration von Anubis. Bitte kontatkiere den Administrator damit dieser die Logs prüfen kann.",
"invalid_redirect": "Ungültige Weiterleitung",
"redirect_not_parseable": "URL der Weiterleitung kann nicht verarbeitet werden",
"redirect_domain_not_allowed": "Domain der Weiterleitung nicht erlaubt",
"failed_to_sign_jwt": "Signierung des JWT fehlgeschlagen",
"invalid_invocation": "Aufrufen von MakeChallenge ungültig",
"client_error_browser": "Client Error: Bitte stelle sicher, dass der Browser aktuell ist und probiere es später erneut.",
"oh_noes": "Vermaledeit!",
"benchmarking_anubis": "Benchmark wird durchgeführt!",
"you_are_not_a_bot": "Sie sind kein Bot!",
"making_sure_not_bot": "Ihr Browser wird geprüft!",
"celphase": "CELPHASE",
"js_web_crypto_error": "Ihr Browser hat leider kein funktionierendes web.crypto Element. Wird eine sichere Verbindung verwendet?",
"js_web_workers_error": "Ihr Browser unterstützt keine Web-Worker (Anubis verwendet diese, damit der Browser nicht unresponsive wird). Ist eventuell ein Plugin wie zB JShelter installiert?",
"js_cookies_error": "Ihr Browser speichert keine Cookies. Anubis verwendet Cookies um ein gültiges Token zu speichern damit es wissen kann, welche Browser bereits geprüft wurden. Bitte aktiviere Cookies für diese Domain. Die Cookie-Namen von Anubis könnten sich jederzeit ändern. Cookie-Namen sind kein Teil der öffentlichen API.",
"js_context_not_secure": "Diese Verbindung ist nicht sicher!",
"js_context_not_secure_msg": "Bitte probiere, dich via HTTPS zu verbinden und lass den Webseiten-Administrator wissen, sauber HTTPS einzurichten. Mehr Informationen unter: <a href=\"https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts#when_is_a_context_considered_secure\">MDN</a>.",
"js_calculating": "Berechnung wird durchgeführt...",
"js_missing_feature": "Fehlendes Feature",
"js_challenge_error": "Fehler während des Checks!",
"js_challenge_error_msg": "Der Check-Algorithmus konnte nicht geladen werden. Bitte lade diese Seite erneut.",
"js_calculating_difficulty": "Berechnung wird durchgeführt...<br/>Schwierigkeit:",
"js_speed": "Geschwindigkeit:",
"js_verification_longer": "Der Check benötigt länger als erwartet. Bitte bleibe auf der Seite.",
"js_success": "Erfolgreich!",
"js_done_took": "Fertig! Dauer:",
"js_iterations": "Iterationen",
"js_finished_reading": "Fertig gelesen, weiter zur Seite →",
"js_calculation_error": "Fehler bei der Berechnung!",
"js_calculation_error_msg": "Fehler bei der Berechnung des Checks:"
}

View File

@@ -1,3 +1,3 @@
{
"supportedLanguages": ["en", "fr", "es", "pt-BR"]
"supportedLanguages": ["en", "fr", "es", "pt-BR", "de", "tr"]
}

View File

@@ -0,0 +1,63 @@
{
"loading": "Yükleniyor...",
"why_am_i_seeing": "Bunu neden görüyorum?",
"protected_by": "Koruma sağlayan:",
"made_with": "🇨🇦'da ❤️ ile yapıldı",
"mascot_design": "Maskot tasarımı:",
"ai_companies_explanation": "Bunu görüyorsunuz çünkü bu web sitesinin yöneticisi, yapay zeka şirketlerinin web sitelerini agresif şekilde kazımasına karşı sunucuyu korumak için Anubis'i kurdu. Bu tarz kazımalar sitelerin erişilemez olmasına ve kesintilere neden olabiliyor.",
"anubis_compromise": "Anubis bir uzlaşmadır. Anubis, spam e-postaları azaltmak için önerilen bir iş kanıtı sistemi olan Hashcash benzeri bir sistemi kullanır. Bireysel kullanımda bu ek yük göz ardı edilebilir olsa da, büyük ölçekli kazıyıcılarda birikerek kazımayı oldukça maliyetli hale getirir.",
"hack_purpose": "Bu geçici bir çözümdür. Esas amacı, başsız tarayıcıları parmak iziyle tanımlamak için daha fazla zaman kazandırmak, ve bu sayede meşru kullanıcıların bu zorluk sayfasını görmesini önlemektir.",
"jshelter_note": "Lütfen dikkat: Anubis, JShelter gibi eklentilerin devre dışı bıraktığı modern JavaScript özelliklerini gerektirir. Lütfen bu alan adı için JShelter veya benzeri eklentileri devre dışı bırakın.",
"version_info": "Bu web sitesi şu Anubis sürümünü çalıştırıyor:",
"try_again": "Tekrar dene",
"go_home": "Ana sayfaya dön",
"contact_webmaster": "ya da engellenmemeniz gerektiğini düşünüyorsanız, lütfen şu adrese e-posta gönderin:",
"connection_security": "Bağlantınızın güvenliği sağlanırken lütfen bekleyin.",
"javascript_required": "Ne yazık ki bu aşamayı geçebilmek için JavaScripti etkinleştirmeniz gerekiyor. Bunun nedeni, yapay zeka şirketlerinin web barındırma konusundaki sosyal sözleşmeyi değiştirmiş olmasıdır. JavaScriptsiz bir çözüm geliştirilmektedir.",
"benchmark_requires_js": "Kıyaslama aracının çalıştırılması için JavaScriptin etkin olması gereklidir.",
"difficulty": "Zorluk:",
"algorithm": "Algoritma:",
"compare": "Karşılaştır:",
"time": "Süre",
"iters": "Tekrar",
"time_a": "Süre A",
"iters_a": "Tekrar A",
"time_b": "Süre B",
"iters_b": "Tekrar B",
"static_check_endpoint": "Bu sadece ters proxy'nizin kullanması için bir kontrol adresidir.",
"authorization_required": "Yetkilendirme gerekli",
"cookies_disabled": "Tarayıcınız çerezleri devre dışı bırakacak şekilde yapılandırılmış. Anubis, gerçek bir kullanıcı olduğunuzu doğrulamak için çerezlere ihtiyaç duyar. Lütfen bu alan adı için çerezleri etkinleştirin.",
"access_denied": "Erişim Reddedildi: hata kodu",
"dronebl_entry": "DroneBL bir giriş bildirdi",
"see_dronebl_lookup": "bakınız",
"internal_server_error": "Sunucu Hatası: Yönetici Anubis'i yanlış yapılandırmış. Lütfen yöneticinizle iletişime geçin ve şunun civarındaki kayıtlara bakmasını isteyin:",
"invalid_redirect": "Geçersiz yönlendirme",
"redirect_not_parseable": "Yönlendirme URL'si çözümlenemiyor",
"redirect_domain_not_allowed": "Yönlendirme alan adına izin verilmiyor",
"failed_to_sign_jwt": "JWT imzalanamadı",
"invalid_invocation": "Geçersiz MakeChallenge çağrısı",
"client_error_browser": "İstemci Hatası: Lütfen tarayıcınızın güncel olduğundan emin olun ve daha sonra tekrar deneyin.",
"oh_noes": "Ah hayır!",
"benchmarking_anubis": "Anubis kıyaslanıyor!",
"you_are_not_a_bot": "Bot değilsiniz!",
"making_sure_not_bot": "Bot olmadığınızdan emin oluyoruz!",
"celphase": "CELPHASE",
"js_web_crypto_error": "Tarayıcınızda çalışan bir web.crypto öğesi yok. Bu sayfayı güvenli bir bağlantı üzerinden mi görüntülüyorsunuz?",
"js_web_workers_error": "Tarayıcınız web işçilerini desteklemiyor (Anubis, tarayıcınızın donmaması için bunları kullanır). JShelter gibi bir eklenti mi kurulu?",
"js_cookies_error": "Tarayıcınız çerezleri kaydetmiyor. Anubis, kullanıcıların zorlukları geçtiğini belirlemek için imzalı bir belirteci çerezde saklar. Lütfen bu alan adı için çerezleri etkinleştirin. Anubisin kullandığı çerez adları önceden bildirilmeksizin değişebilir. Çerez adları ve değerleri resmi APInin bir parçası değildir.",
"js_context_not_secure": "Bağlantınız güvenli değil!",
"js_context_not_secure_msg": "HTTPS üzerinden bağlanmayı deneyin veya yöneticiden HTTPS kurulumu yapmasını isteyin. Daha fazla bilgi için bkz. <a href=\"https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts#when_is_a_context_considered_secure\">MDN</a>.",
"js_calculating": "Hesaplanıyor...",
"js_missing_feature": "Eksik özellik",
"js_challenge_error": "Hesaplama hatası!",
"js_challenge_error_msg": "Algoritma çözümlemesi başarısız oldu. Sayfayı yeniden yüklemeyi deneyebilirsiniz.",
"js_calculating_difficulty": "Hesaplanıyor...<br/>Zorluk:",
"js_speed": "Hız:",
"js_verification_longer": "Doğrulama beklenenden uzun sürüyor. Lütfen sayfayı yenilemeyin.",
"js_success": "Başarılı!",
"js_done_took": "Tamamlandı! Süre:",
"js_iterations": "tekrar",
"js_finished_reading": "Okumayı bitirdim, devam et →",
"js_calculation_error": "Hesaplama hatası!",
"js_calculation_error_msg": "Zorluk hesaplaması başarısız oldu:"
}

View File

@@ -3,6 +3,7 @@ package localization
import (
"embed"
"encoding/json"
"github.com/TecharoHQ/anubis"
"net/http"
"strings"
"sync"
@@ -57,14 +58,14 @@ func NewLocalizationService() *LocalizationService {
globalService = &LocalizationService{bundle: bundle}
})
// Safety check - if globalService is still nil, create a minimal one
if globalService == nil {
bundle := i18n.NewBundle(language.English)
bundle.RegisterUnmarshalFunc("json", json.Unmarshal)
globalService = &LocalizationService{bundle: bundle}
}
return globalService
}
@@ -93,8 +94,13 @@ func (sl *SimpleLocalizer) T(messageID string) string {
return sl.Localizer.MustLocalize(&i18n.LocalizeConfig{MessageID: messageID})
}
// GetLocalizer creates a localizer based on the request's Accept-Language header
// GetLocalizer creates a localizer based on the request's Accept-Language header or forcedLanguage option
func GetLocalizer(r *http.Request) *SimpleLocalizer {
localizer := NewLocalizationService().GetLocalizerFromRequest(r)
var localizer *i18n.Localizer
if anubis.ForcedLanguage == "" {
localizer = NewLocalizationService().GetLocalizerFromRequest(r)
} else {
localizer = NewLocalizationService().GetLocalizer(anubis.ForcedLanguage)
}
return &SimpleLocalizer{Localizer: localizer}
}

View File

@@ -27,6 +27,22 @@ func TestLocalizationService(t *testing.T) {
}
})
t.Run("German localization", func(t *testing.T) {
localizer := service.GetLocalizer("de")
result := localizer.MustLocalize(&i18n.LocalizeConfig{MessageID: "loading"})
if result != "Ladevorgang..." {
t.Errorf("Expected 'Ladevorgang...', got '%s'", result)
}
})
t.Run("Turkish localization", func(t *testing.T) {
localizer := service.GetLocalizer("tr")
result := localizer.MustLocalize(&i18n.LocalizeConfig{MessageID: "loading"})
if result != "Yükleniyor..." {
t.Errorf("Expected 'Yükleniyor...', got '%s'", result)
}
})
t.Run("All required keys exist in English", func(t *testing.T) {
localizer := service.GetLocalizer("en")
requiredKeys := []string{
@@ -56,6 +72,21 @@ func TestLocalizationService(t *testing.T) {
}
}
})
t.Run("All required keys exist in Turkish", func(t *testing.T) {
localizer := service.GetLocalizer("tr")
requiredKeys := []string{
"loading", "why_am_i_seeing", "protected_by", "made_with",
"mascot_design", "try_again", "go_home", "javascript_required",
}
for _, key := range requiredKeys {
result := localizer.MustLocalize(&i18n.LocalizeConfig{MessageID: key})
if result == "" {
t.Errorf("Key '%s' returned empty string", key)
}
}
})
}
type manifest struct {

View File

@@ -329,6 +329,7 @@ type fileConfig struct {
OpenGraph openGraphFileConfig `json:"openGraph,omitempty"`
Impressum *Impressum `json:"impressum,omitempty"`
StatusCodes StatusCodes `json:"status_codes"`
Store *Store `json:"store"`
Thresholds []Threshold `json:"thresholds"`
}
@@ -361,6 +362,12 @@ func (c *fileConfig) Valid() error {
}
}
if c.Store != nil {
if err := c.Store.Valid(); err != nil {
errs = append(errs, err)
}
}
if len(errs) != 0 {
return fmt.Errorf("config is not valid:\n%w", errors.Join(errs...))
}
@@ -374,6 +381,9 @@ func Load(fin io.Reader, fname string) (*Config, error) {
Challenge: http.StatusOK,
Deny: http.StatusOK,
},
Store: &Store{
Backend: "memory",
},
}
if err := yaml.NewYAMLToJSONDecoder(fin).Decode(&c); err != nil {
@@ -392,6 +402,7 @@ func Load(fin io.Reader, fname string) (*Config, error) {
Override: c.OpenGraph.Override,
},
StatusCodes: c.StatusCodes,
Store: c.Store,
}
if c.OpenGraph.TimeToLive != "" {
@@ -457,6 +468,7 @@ type Config struct {
Impressum *Impressum
OpenGraph OpenGraph
StatusCodes StatusCodes
Store *Store
}
func (c Config) Valid() error {

View File

@@ -1,4 +1,4 @@
package config
package config_test
import (
"errors"
@@ -8,6 +8,7 @@ import (
"testing"
"github.com/TecharoHQ/anubis/data"
. "github.com/TecharoHQ/anubis/lib/policy/config"
)
func p[V any](v V) *V { return &v }
@@ -325,37 +326,37 @@ func TestConfigValidBad(t *testing.T) {
func TestBotConfigZero(t *testing.T) {
var b BotConfig
if !b.Zero() {
t.Error("zero value BotConfig is not zero value")
t.Error("zero value config.BotConfig is not zero value")
}
b.Name = "hi"
if b.Zero() {
t.Error("BotConfig with name is zero value")
t.Error("config.BotConfig with name is zero value")
}
b.UserAgentRegex = p(".*")
if b.Zero() {
t.Error("BotConfig with user agent regex is zero value")
t.Error("config.BotConfig with user agent regex is zero value")
}
b.PathRegex = p(".*")
if b.Zero() {
t.Error("BotConfig with path regex is zero value")
t.Error("config.BotConfig with path regex is zero value")
}
b.HeadersRegex = map[string]string{"hi": "there"}
if b.Zero() {
t.Error("BotConfig with headers regex is zero value")
t.Error("config.BotConfig with headers regex is zero value")
}
b.Action = RuleAllow
if b.Zero() {
t.Error("BotConfig with action is zero value")
t.Error("config.BotConfig with action is zero value")
}
b.RemoteAddr = []string{"::/0"}
if b.Zero() {
t.Error("BotConfig with remote addresses is zero value")
t.Error("config.BotConfig with remote addresses is zero value")
}
b.Challenge = &ChallengeRules{
@@ -364,6 +365,6 @@ func TestBotConfigZero(t *testing.T) {
Algorithm: DefaultAlgorithm,
}
if b.Zero() {
t.Error("BotConfig with challenge rules is zero value")
t.Error("config.BotConfig with challenge rules is zero value")
}
}

View File

@@ -0,0 +1,44 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"github.com/TecharoHQ/anubis/lib/store"
_ "github.com/TecharoHQ/anubis/lib/store/all"
)
var (
ErrNoStoreBackend = errors.New("config.Store: no backend defined")
ErrUnknownStoreBackend = errors.New("config.Store: unknown backend")
)
type Store struct {
Backend string `json:"backend"`
Parameters json.RawMessage `json:"parameters"`
}
func (s *Store) Valid() error {
var errs []error
if len(s.Backend) == 0 {
errs = append(errs, ErrNoStoreBackend)
}
fac, ok := store.Get(s.Backend)
switch ok {
case true:
if err := fac.Valid(s.Parameters); err != nil {
errs = append(errs, err)
}
case false:
errs = append(errs, fmt.Errorf("%w: %q", ErrUnknownStoreBackend, s.Backend))
}
if len(errs) != 0 {
return errors.Join(errs...)
}
return nil
}

View File

@@ -0,0 +1,84 @@
package config_test
import (
"encoding/json"
"errors"
"testing"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store/bbolt"
"github.com/TecharoHQ/anubis/lib/store/valkey"
)
func TestStoreValid(t *testing.T) {
for _, tt := range []struct {
name string
input config.Store
err error
}{
{
name: "no backend",
input: config.Store{},
err: config.ErrNoStoreBackend,
},
{
name: "in-memory backend",
input: config.Store{
Backend: "memory",
},
},
{
name: "bbolt backend",
input: config.Store{
Backend: "bbolt",
Parameters: json.RawMessage(`{"path": "/tmp/foo", "bucket": "bar"}`),
},
},
{
name: "valkey backend",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{"url": "redis://valkey:6379/0"}`),
},
},
{
name: "valkey backend no URL",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{}`),
},
err: valkey.ErrNoURL,
},
{
name: "valkey backend bad URL",
input: config.Store{
Backend: "valkey",
Parameters: json.RawMessage(`{"url": "http://anubis.techaro.lol"}`),
},
err: valkey.ErrBadURL,
},
{
name: "bbolt backend no path",
input: config.Store{
Backend: "bbolt",
Parameters: json.RawMessage(`{"path": "", "bucket": "bar"}`),
},
err: bbolt.ErrMissingPath,
},
{
name: "unknown backend",
input: config.Store{
Backend: "taco salad",
},
err: config.ErrUnknownStoreBackend,
},
} {
t.Run(tt.name, func(t *testing.T) {
if err := tt.input.Valid(); !errors.Is(err, tt.err) {
t.Logf("want: %v", tt.err)
t.Logf("got: %v", err)
t.Error("invalid error returned")
}
})
}
}

View File

@@ -11,8 +11,11 @@ import (
"github.com/TecharoHQ/anubis/internal/thoth"
"github.com/TecharoHQ/anubis/lib/policy/checker"
"github.com/TecharoHQ/anubis/lib/policy/config"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
_ "github.com/TecharoHQ/anubis/lib/store/all"
)
var (
@@ -35,9 +38,10 @@ type ParsedConfig struct {
OpenGraph config.OpenGraph
DefaultDifficulty int
StatusCodes config.StatusCodes
Store store.Interface
}
func NewParsedConfig(orig *config.Config) *ParsedConfig {
func newParsedConfig(orig *config.Config) *ParsedConfig {
return &ParsedConfig{
orig: orig,
OpenGraph: orig.OpenGraph,
@@ -55,7 +59,7 @@ func ParseConfig(ctx context.Context, fin io.Reader, fname string, defaultDiffic
tc, hasThothClient := thoth.FromContext(ctx)
result := NewParsedConfig(c)
result := newParsedConfig(c)
result.DefaultDifficulty = defaultDifficulty
for _, b := range c.Bots {
@@ -178,6 +182,19 @@ func ParseConfig(ctx context.Context, fin io.Reader, fname string, defaultDiffic
result.Thresholds = append(result.Thresholds, threshold)
}
stFac, ok := store.Get(c.Store.Backend)
switch ok {
case true:
store, err := stFac.Build(ctx, c.Store.Parameters)
if err != nil {
validationErrs = append(validationErrs, err)
} else {
result.Store = store
}
case false:
validationErrs = append(validationErrs, config.ErrUnknownStoreBackend)
}
if len(validationErrs) > 0 {
return nil, fmt.Errorf("errors validating policy config JSON %s: %w", fname, errors.Join(validationErrs...))
}

10
lib/store/all/all.go Normal file
View File

@@ -0,0 +1,10 @@
// Package all is a meta-package that imports all store implementations.
//
// This is a HACK to make tests work consistently.
package all
import (
_ "github.com/TecharoHQ/anubis/lib/store/bbolt"
_ "github.com/TecharoHQ/anubis/lib/store/memory"
_ "github.com/TecharoHQ/anubis/lib/store/valkey"
)

142
lib/store/bbolt/bbolt.go Normal file
View File

@@ -0,0 +1,142 @@
package bbolt
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"time"
"github.com/TecharoHQ/anubis/lib/store"
"go.etcd.io/bbolt"
)
var (
ErrBucketDoesNotExist = errors.New("bbolt: bucket does not exist")
ErrNotExists = errors.New("bbolt: value does not exist in store")
)
type Item struct {
Data []byte `json:"data"`
Expires time.Time `json:"expires"`
}
type Store struct {
bucket []byte
bdb *bbolt.DB
}
func (s *Store) Delete(ctx context.Context, key string) error {
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
if bkt.Get([]byte(key)) == nil {
return fmt.Errorf("%w: %q", ErrNotExists, key)
}
return bkt.Delete([]byte(key))
})
}
func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
var i Item
if err := s.bdb.View(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
bucketData := bkt.Get([]byte(key))
if bucketData == nil {
return fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
if err := json.Unmarshal(bucketData, &i); err != nil {
return fmt.Errorf("%w: %w", store.ErrCantDecode, err)
}
return nil
}); err != nil {
return nil, err
}
if time.Now().After(i.Expires) {
go s.Delete(context.Background(), key)
return nil, fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return i.Data, nil
}
func (s *Store) Set(ctx context.Context, key string, value []byte, expiry time.Duration) error {
i := Item{
Data: value,
Expires: time.Now().Add(expiry),
}
data, err := json.Marshal(i)
if err != nil {
return fmt.Errorf("%w: %w", store.ErrCantEncode, err)
}
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("%w: %q", ErrBucketDoesNotExist, string(s.bucket))
}
return bkt.Put([]byte(key), data)
})
}
func (s *Store) cleanup(ctx context.Context) error {
now := time.Now()
return s.bdb.Update(func(tx *bbolt.Tx) error {
bkt := tx.Bucket(s.bucket)
if bkt == nil {
return fmt.Errorf("cache bucket %q does not exist", string(s.bucket))
}
return bkt.ForEach(func(k, v []byte) error {
var i Item
data := bkt.Get(k)
if data == nil {
return fmt.Errorf("%s in Cache bucket does not exist???", string(k))
}
if err := json.Unmarshal(data, &i); err != nil {
return fmt.Errorf("can't unmarshal data at key %s: %w", string(k), err)
}
if now.After(i.Expires) {
return bkt.Delete(k)
}
return nil
})
})
}
func (s *Store) cleanupThread(ctx context.Context) {
t := time.NewTicker(time.Hour)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if err := s.cleanup(ctx); err != nil {
slog.Error("error during bbolt cleanup", "err", err)
}
}
}
}

View File

@@ -0,0 +1,23 @@
package bbolt
import (
"encoding/json"
"path/filepath"
"testing"
"github.com/TecharoHQ/anubis/lib/store/storetest"
)
func TestImpl(t *testing.T) {
path := filepath.Join(t.TempDir(), "db")
t.Log(path)
data, err := json.Marshal(Config{
Path: path,
Bucket: "anubis",
})
if err != nil {
t.Fatal(err)
}
storetest.Common(t, Factory{}, json.RawMessage(data))
}

100
lib/store/bbolt/factory.go Normal file
View File

@@ -0,0 +1,100 @@
package bbolt
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"github.com/TecharoHQ/anubis/lib/store"
"go.etcd.io/bbolt"
)
var (
ErrMissingPath = errors.New("bbolt: path is missing from config")
ErrCantWriteToPath = errors.New("bbolt: can't write to path")
)
func init() {
store.Register("bbolt", Factory{})
}
type Factory struct{}
func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface, error) {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if config.Bucket == "" {
config.Bucket = "anubis"
}
bdb, err := bbolt.Open(config.Path, 0600, nil)
if err != nil {
return nil, fmt.Errorf("can't open bbolt database %s: %w", config.Path, err)
}
if err := bdb.Update(func(tx *bbolt.Tx) error {
if _, err := tx.CreateBucketIfNotExists([]byte(config.Bucket)); err != nil {
return err
}
return nil
}); err != nil {
return nil, fmt.Errorf("can't create bbolt bucket %q: %w", config.Bucket, err)
}
result := &Store{
bdb: bdb,
bucket: []byte(config.Bucket),
}
go result.cleanupThread(ctx)
return result, nil
}
func (Factory) Valid(data json.RawMessage) error {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
return nil
}
type Config struct {
Path string `json:"path"`
Bucket string `json:"bucket,omitempty"`
}
func (c Config) Valid() error {
var errs []error
if c.Path == "" {
errs = append(errs, ErrMissingPath)
} else {
dir := filepath.Dir(c.Path)
if err := os.WriteFile(filepath.Join(dir, ".test-file"), []byte(""), 0600); err != nil {
errs = append(errs, ErrCantWriteToPath)
}
}
if len(errs) != 0 {
return errors.Join(errs...)
}
return nil
}

View File

@@ -0,0 +1,42 @@
package bbolt
import (
"encoding/json"
"errors"
"testing"
)
func TestFactoryValid(t *testing.T) {
f := Factory{}
t.Run("bad config", func(t *testing.T) {
if err := f.Valid(json.RawMessage(`}`)); err == nil {
t.Error("wanted parsing failure but got a successful result")
}
})
t.Run("invalid config", func(t *testing.T) {
for _, tt := range []struct {
name string
cfg Config
err error
}{
{
name: "missing path",
cfg: Config{},
err: ErrMissingPath,
},
} {
t.Run(tt.name, func(t *testing.T) {
data, err := json.Marshal(tt.cfg)
if err != nil {
t.Fatal(err)
}
if err := f.Valid(json.RawMessage(data)); !errors.Is(err, tt.err) {
t.Error(err)
}
})
}
})
}

90
lib/store/interface.go Normal file
View File

@@ -0,0 +1,90 @@
package store
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
)
var (
// ErrNotFound is returned when the store implementation cannot find the value
// for a given key.
ErrNotFound = errors.New("store: key not found")
// ErrCantDecode is returned when a store adaptor cannot decode the store format
// to a value used by the code.
ErrCantDecode = errors.New("store: can't decode value")
// ErrCantEncode is returned when a store adaptor cannot encode the value into
// the format that the store uses.
ErrCantEncode = errors.New("store: can't encode value")
// ErrBadConfig is returned when a store adaptor's configuration is invalid.
ErrBadConfig = errors.New("store: configuration is invalid")
)
// Interface defines the calls that Anubis uses for storage in a local or remote
// datastore. This can be implemented with an in-memory, on-disk, or in-database
// storage backend.
type Interface interface {
// Delete removes a value from the store by key.
Delete(ctx context.Context, key string) error
// Get returns the value of a key assuming that value exists and has not expired.
Get(ctx context.Context, key string) ([]byte, error)
// Set puts a value into the store that expires according to its expiry.
Set(ctx context.Context, key string, value []byte, expiry time.Duration) error
}
func z[T any]() T { return *new(T) }
type JSON[T any] struct {
Underlying Interface
Prefix string
}
func (j *JSON[T]) Delete(ctx context.Context, key string) error {
if j.Prefix != "" {
key = j.Prefix + key
}
return j.Underlying.Delete(ctx, key)
}
func (j *JSON[T]) Get(ctx context.Context, key string) (T, error) {
if j.Prefix != "" {
key = j.Prefix + key
}
data, err := j.Underlying.Get(ctx, key)
if err != nil {
return z[T](), err
}
var result T
if err := json.Unmarshal(data, &result); err != nil {
return z[T](), fmt.Errorf("%w: %w", ErrCantDecode, err)
}
return result, nil
}
func (j *JSON[T]) Set(ctx context.Context, key string, value T, expiry time.Duration) error {
if j.Prefix != "" {
key = j.Prefix + key
}
data, err := json.Marshal(value)
if err != nil {
return fmt.Errorf("%w: %w", ErrCantEncode, err)
}
if err := j.Underlying.Set(ctx, key, data, expiry); err != nil {
return err
}
return nil
}

50
lib/store/json_test.go Normal file
View File

@@ -0,0 +1,50 @@
package store_test
import (
"testing"
"time"
"github.com/TecharoHQ/anubis/lib/store"
"github.com/TecharoHQ/anubis/lib/store/memory"
)
func TestJSON(t *testing.T) {
type data struct {
ID string `json:"id"`
}
st := memory.New(t.Context())
db := store.JSON[data]{
Underlying: st,
Prefix: "foo:",
}
if err := db.Set(t.Context(), "test", data{ID: t.Name()}, time.Minute); err != nil {
t.Fatal(err)
}
got, err := db.Get(t.Context(), "test")
if err != nil {
t.Fatal(err)
}
if got.ID != t.Name() {
t.Fatalf("got wrong data for key \"test\", wanted %q but got: %q", t.Name(), got.ID)
}
if err := db.Delete(t.Context(), "test"); err != nil {
t.Fatal(err)
}
if _, err := db.Get(t.Context(), "test"); err == nil {
t.Fatal("wanted invalid get to fail, it did not")
}
if err := st.Set(t.Context(), "foo:test", []byte("}"), time.Minute); err != nil {
t.Fatal(err)
}
if _, err := db.Get(t.Context(), "test"); err == nil {
t.Fatal("wanted invalid get to fail, it did not")
}
}

View File

@@ -0,0 +1,74 @@
package memory
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/TecharoHQ/anubis/decaymap"
"github.com/TecharoHQ/anubis/lib/store"
)
type factory struct{}
func (factory) Build(ctx context.Context, _ json.RawMessage) (store.Interface, error) {
return New(ctx), nil
}
func (factory) Valid(json.RawMessage) error { return nil }
func init() {
store.Register("memory", factory{})
}
type impl struct {
store *decaymap.Impl[string, []byte]
}
func (i *impl) Delete(_ context.Context, key string) error {
if !i.store.Delete(key) {
return fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return nil
}
func (i *impl) Get(_ context.Context, key string) ([]byte, error) {
result, ok := i.store.Get(key)
if !ok {
return nil, fmt.Errorf("%w: %q", store.ErrNotFound, key)
}
return result, nil
}
func (i *impl) Set(_ context.Context, key string, value []byte, expiry time.Duration) error {
i.store.Set(key, value, expiry)
return nil
}
func (i *impl) cleanupThread(ctx context.Context) {
t := time.NewTicker(5 * time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
i.store.Cleanup()
}
}
}
// New creates a simple in-memory store. This will not scale to multiple Anubis instances.
func New(ctx context.Context) store.Interface {
result := &impl{
store: decaymap.New[string, []byte](),
}
go result.cleanupThread(ctx)
return result
}

View File

@@ -0,0 +1,11 @@
package memory
import (
"testing"
"github.com/TecharoHQ/anubis/lib/store/storetest"
)
func TestImpl(t *testing.T) {
storetest.Common(t, factory{}, nil)
}

43
lib/store/registry.go Normal file
View File

@@ -0,0 +1,43 @@
package store
import (
"context"
"encoding/json"
"sort"
"sync"
)
var (
registry map[string]Factory = map[string]Factory{}
regLock sync.RWMutex
)
type Factory interface {
Build(ctx context.Context, config json.RawMessage) (Interface, error)
Valid(config json.RawMessage) error
}
func Register(name string, impl Factory) {
regLock.Lock()
defer regLock.Unlock()
registry[name] = impl
}
func Get(name string) (Factory, bool) {
regLock.RLock()
defer regLock.RUnlock()
result, ok := registry[name]
return result, ok
}
func Methods() []string {
regLock.RLock()
defer regLock.RUnlock()
var result []string
for method := range registry {
result = append(result, method)
}
sort.Strings(result)
return result
}

View File

@@ -0,0 +1,92 @@
package storetest
import (
"bytes"
"encoding/json"
"errors"
"testing"
"time"
"github.com/TecharoHQ/anubis/lib/store"
)
func Common(t *testing.T, f store.Factory, config json.RawMessage) {
if err := f.Valid(config); err != nil {
t.Fatal(err)
}
s, err := f.Build(t.Context(), config)
if err != nil {
t.Fatal(err)
}
for _, tt := range []struct {
name string
doer func(t *testing.T, s store.Interface) error
err error
}{
{
name: "basic get set delete",
doer: func(t *testing.T, s store.Interface) error {
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to not exist in store but it exists anyways", t.Name())
}
if err := s.Set(t.Context(), t.Name(), []byte(t.Name()), 5*time.Minute); err != nil {
return err
}
val, err := s.Get(t.Context(), t.Name())
if errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to exist in store but it does not", t.Name())
}
if !bytes.Equal(val, []byte(t.Name())) {
t.Logf("want: %q", t.Name())
t.Logf("got: %q", string(val))
t.Error("wrong value returned")
}
if err := s.Delete(t.Context(), t.Name()); err != nil {
return err
}
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Error("wanted test to not exist in store but it exists anyways")
}
if err := s.Delete(t.Context(), t.Name()); err == nil {
t.Errorf("key %q does not exist and Delete did not return non-nil", t.Name())
}
return nil
},
},
{
name: "expires",
doer: func(t *testing.T, s store.Interface) error {
if err := s.Set(t.Context(), t.Name(), []byte(t.Name()), 150*time.Millisecond); err != nil {
return err
}
//nosleep:bypass XXX(Xe): use Go's time faking thing in Go 1.25 when that is released.
time.Sleep(155 * time.Millisecond)
if _, err := s.Get(t.Context(), t.Name()); !errors.Is(err, store.ErrNotFound) {
t.Errorf("wanted %s to not exist in store but it exists anyways", t.Name())
}
return nil
},
},
} {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if err := tt.doer(t, s); !errors.Is(err, tt.err) {
t.Logf("want: %v", tt.err)
t.Logf("got: %v", err)
t.Error("wrong error")
}
})
}
}

View File

@@ -0,0 +1,84 @@
package valkey
import (
"context"
"encoding/json"
"errors"
"fmt"
"github.com/TecharoHQ/anubis/lib/store"
valkey "github.com/redis/go-redis/v9"
)
var (
ErrNoURL = errors.New("valkey.Config: no URL defined")
ErrBadURL = errors.New("valkey.Config: URL is invalid")
)
func init() {
store.Register("valkey", Factory{})
}
type Factory struct{}
func (Factory) Build(ctx context.Context, data json.RawMessage) (store.Interface, error) {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
opts, err := valkey.ParseURL(config.URL)
if err != nil {
return nil, fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
rdb := valkey.NewClient(opts)
if _, err := rdb.Ping(ctx).Result(); err != nil {
return nil, fmt.Errorf("can't ping valkey instance: %w", err)
}
return &Store{
rdb: rdb,
}, nil
}
func (Factory) Valid(data json.RawMessage) error {
var config Config
if err := json.Unmarshal([]byte(data), &config); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
if err := config.Valid(); err != nil {
return fmt.Errorf("%w: %w", store.ErrBadConfig, err)
}
return nil
}
type Config struct {
URL string `json:"url"`
}
func (c Config) Valid() error {
var errs []error
if c.URL == "" {
errs = append(errs, ErrNoURL)
}
if _, err := valkey.ParseURL(c.URL); err != nil {
errs = append(errs, ErrBadURL)
}
if len(errs) != 0 {
return fmt.Errorf("valkey.Config: invalid config: %w", errors.Join(errs...))
}
return nil
}

View File

@@ -0,0 +1,49 @@
package valkey
import (
"context"
"fmt"
"time"
"github.com/TecharoHQ/anubis/lib/store"
valkey "github.com/redis/go-redis/v9"
)
type Store struct {
rdb *valkey.Client
}
func (s *Store) Delete(ctx context.Context, key string) error {
n, err := s.rdb.Del(ctx, key).Result()
if err != nil {
return fmt.Errorf("can't delete from valkey: %w", err)
}
switch n {
case 0:
return fmt.Errorf("%w: %d key(s) deleted", store.ErrNotFound, n)
default:
return nil
}
}
func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
result, err := s.rdb.Get(ctx, key).Result()
if err != nil {
if valkey.HasErrorPrefix(err, "redis: nil") {
return nil, fmt.Errorf("%w: %w", store.ErrNotFound, err)
}
return nil, fmt.Errorf("can't fetch from valkey: %w", err)
}
return []byte(result), nil
}
func (s *Store) Set(ctx context.Context, key string, value []byte, expiry time.Duration) error {
if _, err := s.rdb.Set(ctx, key, string(value), expiry).Result(); err != nil {
return fmt.Errorf("can't set %q in valkey: %w", key, err)
}
return nil
}

View File

@@ -0,0 +1,53 @@
package valkey
import (
"encoding/json"
"fmt"
"os"
"testing"
"github.com/TecharoHQ/anubis/internal"
"github.com/TecharoHQ/anubis/lib/store/storetest"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)
func init() {
internal.UnbreakDocker()
}
func TestImpl(t *testing.T) {
if os.Getenv("DONT_USE_NETWORK") != "" {
t.Skip("test requires network egress")
return
}
testcontainers.SkipIfProviderIsNotHealthy(t)
req := testcontainers.ContainerRequest{
Image: "valkey/valkey:8",
WaitingFor: wait.ForLog("Ready to accept connections"),
}
valkeyC, err := testcontainers.GenericContainer(t.Context(), testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
})
testcontainers.CleanupContainer(t, valkeyC)
if err != nil {
t.Fatal(err)
}
containerIP, err := valkeyC.ContainerIP(t.Context())
if err != nil {
t.Fatal(err)
}
data, err := json.Marshal(Config{
URL: fmt.Sprintf("redis://%s:6379/0", containerIP),
})
if err != nil {
t.Fatal(err)
}
storetest.Common(t, Factory{}, json.RawMessage(data))
}

View File

@@ -12,7 +12,7 @@ CacheDirectory=anubis/%i
CacheDirectoryMode=0755
StateDirectory=anubis/%i
StateDirectoryMode=0755
RuntimeDirectory=anubis
RuntimeDirectory=anubis/%i
RuntimeDirectoryMode=0755
ReadWritePaths=/run

2
web/index_templ.go generated
View File

@@ -1,6 +1,6 @@
// Code generated by templ - DO NOT EDIT.
// templ: version: v0.3.898
// templ: version: v0.3.906
package web
//lint:file-ignore SA4006 This context is only used if a nested component is present.

View File

@@ -212,11 +212,6 @@ const t = (key) => translations[`js_${key}`] || translations[key] || key;
const t1 = Date.now();
console.log({ hash, nonce });
title.innerHTML = t('success');
status.innerHTML = `${t('done_took')} ${t1 - t0}ms, ${nonce} ${t('iterations')}`;
image.src = imageURL("happy", anubisVersion, basePrefix);
progress.style.display = "none";
if (userReadDetails) {
const container = document.getElementById("progress");
@@ -251,17 +246,15 @@ const t = (key) => translations[`js_${key}`] || translations[key] || key;
container.onclick = onDetailsExpand;
setTimeout(onDetailsExpand, 30000);
} else {
setTimeout(() => {
const redir = window.location.href;
window.location.replace(
u(`${basePrefix}/.within.website/x/cmd/anubis/api/pass-challenge`, {
response: hash,
nonce,
redir,
elapsedTime: t1 - t0,
}),
);
}, 250);
const redir = window.location.href;
window.location.replace(
u(`${basePrefix}/.within.website/x/cmd/anubis/api/pass-challenge`, {
response: hash,
nonce,
redir,
elapsedTime: t1 - t0,
}),
);
}
} catch (err) {
ohNoes({

View File

@@ -9,7 +9,6 @@ User-agent: Applebot-Extended
User-agent: bedrockbot
User-agent: Brightbot 1.0
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User