arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-22 16:16:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(data): add Meta's web indexer used for AI purposes (#1573)

Browse Source 
This indexer is documented in
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. I
saw it parsing the entirety of my Forgejo instance, so I suggest to
widely block it.

Signed-off-by: Benjamin Bouvier <benjamin@bouvier.cc>
This commit is contained in:
Benjamin Bouvier
2026-04-21 22:56:23 +02:00
committed by GitHub
parent d5ccf9c670
commit f21706eb12
2 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/crawlers/ai-search.yaml
+1 -1
View File
@@ -4,5 +4,5 @@
# - Claude-SearchBot: No published IP allowlist
- name: "ai-crawlers-search"
user_agent_regex: >-
OAI-SearchBot|Claude-SearchBot|PerplexityBot
OAI-SearchBot|Claude-SearchBot|PerplexityBot|meta-webindexer
action: DENY