mirror of
https://github.com/TecharoHQ/anubis.git
synced 2026-04-23 00:26:42 +00:00
Benjamin Bouvier
f21706eb12
This indexer is documented in https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. I saw it parsing the entirety of my Forgejo instance, so I suggest to widely block it. Signed-off-by: Benjamin Bouvier <benjamin@bouvier.cc>
9 lines
355 B
YAML
9 lines
355 B
YAML
# User agents that index exclusively for search in for AI systems.
|
|
# Each entry should have a positive/ALLOW entry created as well, with further documentation.
|
|
# Exceptions:
|
|
# - Claude-SearchBot: No published IP allowlist
|
|
- name: "ai-crawlers-search"
|
|
user_agent_regex: >-
|
|
OAI-SearchBot|Claude-SearchBot|PerplexityBot|meta-webindexer
|
|
action: DENY
|