mirror of
https://github.com/TecharoHQ/anubis.git
synced 2026-05-09 08:32:51 +00:00
Xe Iaso
c460169047
Most of the worst of the worst scrapers run Headless Chrome. Headless Chrome is difficult for Anubis to combat because it follows all the rules that browsers do. The worst of the worst scrapers also use residential proxy services. Those residental proxy services charge upwards of $1 per GB of data egressed or ingressed. The Prompt API makes Chrome download a 4Gi or 16Gi machine learning model. When you ask it to start downloading, it will _continue_ downloading even when you leave the Anubis challenge page. This will make the local model answer "why is the sky blue?" in an absurt amount of detail, which wastes both bandwidth and scraper CPU (some scraping companies charge via Chrome CPU too). Signed-off-by: Xe Iaso <me@xeiaso.net>
Website
This website is built using Docusaurus, a modern static website generator.
Installation
$ yarn
Local Development
$ yarn start
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
Build
$ yarn build
This command generates static content into the build directory and can be served using any static contents hosting service.
Deployment
Using SSH:
$ USE_SSH=true yarn deploy
Not using SSH:
$ GIT_USER=<Your GitHub username> yarn deploy
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.