arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-10 18:48:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(default-config): allowlist common crawl

Browse Source 
This may seem strange, but allowlisting common crawl means that scrapers
have less incentive to scrape because they can just grab the data from
common crawl instead of scraping it again.
This commit is contained in:
Xe Iaso
2025-07-04 00:03:54 +00:00
parent d7a758f805
commit da8ebae94e
5 changed files with 19 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/docs/CHANGELOG.md
View File

@@ -23,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Add translation for German language ([#741](https://github.com/TecharoHQ/anubis/pull/741))
- Remove the "Success" interstitial after a proof of work challenge is concluded.
- Add option for forcing a specific language ([#742](https://github.com/TecharoHQ/anubis/pull/742))
- Allow [Common Crawl](https://commoncrawl.org/) by default so scrapers have less incentive to scrape
### Potentially breaking changes