arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-15 04:58:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bump ai.robots.txt to v1.37 (#689)

Browse Source 
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
This commit is contained in:
Dryusdan
2025-06-18 19:30:53 +02:00
committed by GitHub
parent 97c1d4f353
commit 1d5fa49eb0
3 changed files with 11 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
web/static/robots.txt
View File

@@ -21,7 +21,9 @@ User-agent: Cotoyogi
User-agent: Crawlspace
User-agent: Diffbot
User-agent: DuckAssistBot
User-agent: EchoboxBot
User-agent: FacebookBot
User-agent: facebookexternalhit
User-agent: Factset_spyderbot
User-agent: FirecrawlAgent
User-agent: FriendlyCrawler
@@ -42,6 +44,7 @@ User-agent: Meta-ExternalAgent
User-agent: meta-externalfetcher
User-agent: Meta-ExternalFetcher
User-agent: MistralAI-User/1.0
User-agent: MyCentralAIScraperBot
User-agent: NovaAct
User-agent: OAI-SearchBot
User-agent: omgili
@@ -54,12 +57,17 @@ User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: Poseidon Research Crawler
User-agent: QualifiedBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: SBIntuitionsBot
User-agent: Scrapy
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-CT
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: Sidetrade indexer bot
User-agent: TikTokSpider