arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-28 02:52:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Applebot definition (#589)

Browse Source 
* Add Applebot definition

Adds Apple's search indexing bot, and allowlists it by default.

Allowlisted by default because it is equivalent to Googlebot/Bingbot. Remove Applebot from `ai-robots-txt.yaml` for the same reasons.

Remove `Applebot-Extended` from `ai-robots-txt.yaml` as it has no effect.

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Corry Haines
2025-05-31 07:18:32 -07:00
committed by GitHub
parent fbbab5a035
commit 68a71c6a99
6 changed files with 28 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
data/crawlers/_allow-good.yaml
+1
View File
@@ -1,4 +1,5 @@
- import: (data)/crawlers/googlebot.yaml
- import: (data)/crawlers/applebot.yaml
- import: (data)/crawlers/bingbot.yaml
- import: (data)/crawlers/duckduckbot.yaml
- import: (data)/crawlers/qwantbot.yaml