Dryusdan 237a6a98e2 Bump ai.robots.txt to v1.39 (#982)

2025-08-18 06:52:23 -04:00

..

irc-bots

fix(bots/phrik): add IPv6 address for phrik (#494)

2025-05-11 14:04:44 -04:00

_deny-pathological.yaml

feat(default-rules): add weight to Custom-AsyncHttpClient (#914)

2025-07-27 00:41:43 +00:00

aggressive-brazilian-scrapers.yaml

feat(lib): implement request weight (#621)

2025-06-09 15:25:04 -04:00

ai-catchall.yaml

chore(default-config): allowlist common crawl (#753)

2025-07-04 00:10:45 +00:00

ai-robots-txt.yaml

Bump ai.robots.txt to v1.39 (#982)

2025-08-18 06:52:23 -04:00

cloudflare-workers.yaml

feat(lib): implement request weight (#621)

2025-06-09 15:25:04 -04:00

custom-async-http-client.yaml

feat(default-rules): add weight to Custom-AsyncHttpClient (#914)

2025-07-27 00:41:43 +00:00

headless-browsers.yaml

feat: enable loading config fragments (#321)

2025-04-23 07:01:28 -04:00

us-ai-scraper.yaml

feat: enable loading config fragments (#321)

2025-04-23 07:01:28 -04:00