Benjamin Bouvier f21706eb12 feat(data): add Meta's web indexer used for AI purposes (#1573) ...

This indexer is documented in
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. I
saw it parsing the entirety of my Forgejo instance, so I suggest to
widely block it.

Signed-off-by: Benjamin Bouvier <benjamin@bouvier.cc>

2026-04-21 16:56:23 -04:00

..

_allow-good.yaml

feat(data): add Citoid to good bots list (#1524)

2026-03-20 11:13:26 +00:00

ai-search.yaml

feat(data): add Meta's web indexer used for AI purposes (#1573)

2026-04-21 16:56:23 -04:00

ai-training.yaml

Split up AI filtering files (#592)

2025-06-01 20:21:18 +00:00

alibaba-cloud.yaml

fix(default-config): also block alibaba cloud (#1005)

2025-08-20 23:01:49 +00:00

applebot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

bingbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

commoncrawl.yaml

chore(default-config): allowlist common crawl (#753)

2025-07-04 00:10:45 +00:00

duckduckbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

googlebot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

huawei-cloud.yaml

fix(default-config): block Huawei Cloud (#1004)

2025-08-20 22:40:07 +00:00

internet-archive.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

kagibot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

marginalia.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

mojeekbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

openai-gptbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

openai-searchbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

perplexitybot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

qwantbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00

tencent-cloud.yaml

feat(default-config): block tencent cloud by default (#1216)

2025-10-24 19:43:42 +00:00

wikimedia-citoid.yaml

feat(data): add Citoid to good bots list (#1524)

2026-03-20 11:13:26 +00:00

yandexbot.yaml

chore: set up commitlint, husky, and prettier (#1451)

2026-02-15 08:19:12 -05:00