Xe Iaso 7c0996448a chore(default-config): allowlist common crawl (#753) ...

This may seem strange, but allowlisting common crawl means that scrapers
have less incentive to scrape because they can just grab the data from
common crawl instead of scraping it again.

2025-07-04 00:10:45 +00:00

..

irc-bots

fix(bots/phrik): add IPv6 address for phrik (#494)

2025-05-11 14:04:44 -04:00

_deny-pathological.yaml

feat(config): allow multi-level imports (#402)

2025-05-02 13:57:20 -04:00

aggressive-brazilian-scrapers.yaml

feat(lib): implement request weight (#621)

2025-06-09 15:25:04 -04:00

ai-catchall.yaml

chore(default-config): allowlist common crawl (#753)

2025-07-04 00:10:45 +00:00

ai-robots-txt.yaml

chore(default-config): allowlist common crawl (#753)

2025-07-04 00:10:45 +00:00

cloudflare-workers.yaml

feat(lib): implement request weight (#621)

2025-06-09 15:25:04 -04:00

headless-browsers.yaml

feat: enable loading config fragments (#321)

2025-04-23 07:01:28 -04:00

us-ai-scraper.yaml

feat: enable loading config fragments (#321)

2025-04-23 07:01:28 -04:00