Compare commits

..

5 Commits

Author SHA1 Message Date
Xe Iaso e9969ba22a chore: update spelling
check-spelling run (pull_request) for Xe/ci-multiple-go-versions

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
2026-01-16 11:21:06 -05:00
Xe Iaso 7db2c9ebb5 ci: test against multiple go versions
Several Linux distributions and other open source package managers
build Anubis against Go oldstable. Eventually this will have to expand
to at least what FreeBSD supports. I hope they can bump the versions
of Go as soon as upstream makes new code available.

Signed-off-by: Xe Iaso <me@xeiaso.net>
2026-01-16 11:18:31 -05:00
Timon de Groot 57c0b2b22c Add IP mapped Perplexity user agents (#1393)
Perplexity has some proper documentation available for their crawlers,
with published IP addresses: https://docs.perplexity.ai/guides/bots.

Signed-off-by: Timon de Groot <timon.degroot@team.blue>
2026-01-15 19:57:31 -05:00
Thomas Arrow 186ffeb744 docs: clarify botstopper kubernetes instructions (#1404)
This makes it clear that when generating a kubernetes secret to pull the bot stopper image that:
- no email is required
- a user is required but the actual value of the username is not checked
- the GH token needs to be pasted in

Signed-off-by: Thomas Arrow <tarrow@users.noreply.github.com>
2026-01-15 11:13:10 +00:00
Xe Iaso ff87aac4e7 fix(web): include base prefix in generated URLs (#1403)
* fix(web): include base prefix in generated URLs

Forgot to add the base prefix to these URLs. Committed a fix for this
and added a test to ensure this does not repeat. Oops!

Closes: #1402
Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2026-01-14 23:47:44 +00:00
9 changed files with 48 additions and 10 deletions
+2 -1
View File
@@ -64,7 +64,6 @@ ckie
cloudflare cloudflare
Codespaces Codespaces
confd confd
connnection
containerbuild containerbuild
containerregistry containerregistry
coreutils coreutils
@@ -243,6 +242,7 @@ oci
OCOB OCOB
ogtag ogtag
oklch oklch
oldstable
omgili omgili
omgilibot omgilibot
openai openai
@@ -257,6 +257,7 @@ Pangu
parseable parseable
passthrough passthrough
Patreon Patreon
perplexitybot
pgrep pgrep
phrik phrik
pidfile pidfile
+8 -2
View File
@@ -12,6 +12,11 @@ permissions:
jobs: jobs:
go_tests: go_tests:
strategy:
matrix:
go_version:
- oldstable
- stable
#runs-on: alrest-techarohq #runs-on: alrest-techarohq
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
@@ -26,10 +31,11 @@ jobs:
- uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0 - uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0
with: with:
node-version: '24.11.0' node-version: "latest"
- uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 - uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0
with: with:
go-version: '1.25.4' go-version: ${{ matrix.go_version }}
- name: Cache playwright binaries - name: Cache playwright binaries
uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1 uses: actions/cache@9255dc7a253b0ccc959486e2bca901246202afeb # v5.0.1
+1 -1
View File
@@ -4,5 +4,5 @@
# - Claude-User: No published IP allowlist # - Claude-User: No published IP allowlist
- name: "ai-clients" - name: "ai-clients"
user_agent_regex: >- user_agent_regex: >-
ChatGPT-User|Claude-User|MistralAI-User ChatGPT-User|Claude-User|MistralAI-User|Perplexity-User
action: DENY action: DENY
+12
View File
@@ -0,0 +1,12 @@
# Acts on behalf of user requests
# https://docs.perplexity.ai/guides/bots
- name: perplexity-user
user_agent_regex: Perplexity-User/.+; \+https\://perplexity\.ai/perplexity-user
action: ALLOW
# https://www.perplexity.com/perplexity-user.json
remote_addresses: [
"44.208.221.197/32",
"34.193.163.52/32",
"18.97.21.0/30",
"18.97.43.80/29",
]
+1 -1
View File
@@ -4,5 +4,5 @@
# - Claude-SearchBot: No published IP allowlist # - Claude-SearchBot: No published IP allowlist
- name: "ai-crawlers-search" - name: "ai-crawlers-search"
user_agent_regex: >- user_agent_regex: >-
OAI-SearchBot|Claude-SearchBot OAI-SearchBot|Claude-SearchBot|PerplexityBot
action: DENY action: DENY
+16
View File
@@ -0,0 +1,16 @@
# Indexing for search, does not collect training data
# https://docs.perplexity.ai/guides/bots
- name: perplexitybot
user_agent_regex: PerplexityBot/.+; \+https\://perplexity\.ai/perplexitybot
action: ALLOW
# https://www.perplexity.com/perplexitybot.json
remote_addresses: [
"107.20.236.150/32",
"3.224.62.45/32",
"18.210.92.235/32",
"3.222.232.239/32",
"3.211.124.183/32",
"3.231.139.107/32",
"18.97.1.228/30",
"18.97.9.96/29",
]
+3 -1
View File
@@ -3,5 +3,7 @@
- import: (data)/bots/ai-catchall.yaml - import: (data)/bots/ai-catchall.yaml
- import: (data)/crawlers/ai-training.yaml - import: (data)/crawlers/ai-training.yaml
- import: (data)/crawlers/openai-searchbot.yaml - import: (data)/crawlers/openai-searchbot.yaml
- import: (data)/crawlers/perplexitybot.yaml
- import: (data)/clients/openai-chatgpt-user.yaml - import: (data)/clients/openai-chatgpt-user.yaml
- import: (data)/clients/mistral-mistralai-user.yaml - import: (data)/clients/mistral-mistralai-user.yaml
- import: (data)/clients/perplexity-user.yaml
+3 -1
View File
@@ -2,5 +2,7 @@
- import: (data)/bots/ai-catchall.yaml - import: (data)/bots/ai-catchall.yaml
- import: (data)/crawlers/openai-searchbot.yaml - import: (data)/crawlers/openai-searchbot.yaml
- import: (data)/crawlers/openai-gptbot.yaml - import: (data)/crawlers/openai-gptbot.yaml
- import: (data)/crawlers/perplexitybot.yaml
- import: (data)/clients/openai-chatgpt-user.yaml - import: (data)/clients/openai-chatgpt-user.yaml
- import: (data)/clients/mistral-mistralai-user.yaml - import: (data)/clients/mistral-mistralai-user.yaml
- import: (data)/clients/perplexity-user.yaml
+2 -3
View File
@@ -51,9 +51,8 @@ If you are using Kubernetes, you will need to create an image pull secret:
kubectl create secret docker-registry \ kubectl create secret docker-registry \
techarohq-botstopper \ techarohq-botstopper \
--docker-server ghcr.io \ --docker-server ghcr.io \
--docker-username your-username \ --docker-username any-username \
--docker-password your-access-token \ --docker-password <your-access-token> \
--docker-email your@email.address
``` ```
Then attach it to your Deployment: Then attach it to your Deployment: