arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-14 04:28:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bump AI-robots.txt rules to version 1.30 (#509)

Browse Source 
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Dryusdan
2025-05-16 18:40:25 +02:00
committed by GitHub
parent 91c21fbb4b
commit 961320540b
3 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
web/static/robots.txt
View File

@@ -46,6 +46,7 @@ User-agent: PanguBot
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: QualifiedBot
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA