Commit Graph

345 Commits

Author SHA1 Message Date
Xe Iaso
fdb2bb755d fix(yeetfile): put robots2policy in $PATH
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-13 15:34:58 -04:00
Xe Iaso
53bacc7ff4 chore(yeetfile): expose robots2policy in libexec
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-13 15:18:15 -04:00
Xe Iaso
95fccd2966 feat(config): properly marshal bot policy rules
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-13 15:14:39 -04:00
Xe Iaso
332320791a fix(robots2policy): use sigs.k8s.io/yaml
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-13 14:40:23 -04:00
Jason Cameron
00cdc52e9e Merge branch 'main' into json/robots2policycli 2025-06-13 14:26:17 -04:00
Jason Cameron
2dd2a2fa8c Update cmd/robots2policy/main.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
2025-06-13 14:26:04 -04:00
Jason Cameron
31c0351c04 Update cmd/robots2policy/main.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
2025-06-13 14:25:54 -04:00
Jason Cameron
7a74dddb3e docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI 2025-06-13 14:14:50 -04:00
Jason Cameron
995abcbe68 Update metadata
check-spelling run (pull_request) for json/robots2policycli

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
2025-06-13 13:44:39 -04:00
Jason Cameron
c48029b48e fix: update example URL in usage instructions for robots2policy CLI 2025-06-13 13:44:05 -04:00
Jason Cameron
d1947cadd2 test: add comprehensive tests for robots2policy conversion and parsing 2025-06-13 13:40:49 -04:00
Jason Cameron
714496e317 feat: add various robots.txt and YAML configurations for user agent handling and crawl delays 2025-06-13 13:40:49 -04:00
Jason Cameron
5a85cd41a2 feat: implement crawl delay handling as weight adjustment in Anubis rules 2025-06-13 13:40:48 -04:00
Jason Cameron
357b4ba6d5 feat: add documentation for robots2policy CLI tool 2025-06-13 13:40:48 -04:00
Jason Cameron
9ad2d42b9f feat: add robots2policy CLI utility to convert robots.txt to Anubis challenge policies 2025-06-13 13:40:48 -04:00
Jason Cameron
2904ff974b refactor(ogtags): optimize URL construction and memory allocations (#647)
* refactor(ogtags): optimize URL construction and memory allocations

* test(ogtags): add benchmarks and memory usage tests for OGTagCache

* refactor(ogtags): optimize OGTags subsystem to reduce allocations and improve request runtime by up to 66%

* Update docs/docs/CHANGELOG.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* refactor(ogtags): optimize URL string construction to reduce allocations

* Update internal/ogtags/ogtags.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* test(ogtags): add fuzz tests for getTarget and extractOGTags functions

* fix(ogtags): update memory calculation logic

Prev it would say that we had allocated 18pb

=== RUN   TestMemoryUsage
    mem_test.go:107: Memory allocated for 10k getTarget calls: 18014398509481904.00 KB
    mem_test.go:135: Memory allocated for 1k extractOGTags calls: 18014398509481978.00

    Now it's fixed with

    === RUN   TestMemoryUsage
    mem_test.go:109: Memory allocated for 10k getTarget calls:
    mem_test.go:110:   Total: 630.56 KB (0.62 MB)
    mem_test.go:111:   Per operation: 64.57 bytes
    mem_test.go:140: Memory allocated for 1k extractOGTags calls:
    mem_test.go:141:   Total: 328.17 KB (0.32 MB)
    mem_test.go:142:   Per operation: 336.05 bytes

* refactor(ogtags): optimize meta tag extraction for improved performance

* Update metadata

check-spelling run (pull_request) for json/ogmem

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* chore: update CHANGELOG for recent optimizations and version bump

* refactor: improve URL construction and meta tag extraction logic

* style:  cleanup fuzz tests

---------

Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-13 09:53:10 -04:00
Jason Cameron
3b3080d497 feat: add a strip-base-prefix option (#655)
* style: fix formatting in .air.toml and installation.mdx

* feat: add --strip-base-prefix flag to modify request paths when forwarding

Closes: #638

* refactor: apply structpacking (betteralign)

* fix: add validation for strip-base-prefix and base-prefix configuration

* fix: improve request path handling by cloning request and modifying URL path

* chore: remove integration tests as they are too annoying to debug on my system
2025-06-12 17:46:08 -04:00
Jason Cameron
60ba8e9557 fix(ci): conditionally run SSH jobs for TecharoHQ/anubis repository (#654)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-11 21:18:43 +00:00
Jason Cameron
14c80483a9 fix(gitattributes): update pattern for generated files (#652)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-11 21:00:37 +00:00
Xe Iaso
d1452b6d39 test(ssh-ci): re-enable GOARCH=ppc64le (#651)
This reverts commit 5e95da6b6c.
2025-06-11 14:01:48 -04:00
Xe Iaso
5e95da6b6c test(ssh-ci): disable GOARCH=ppc64le for now
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-11 12:58:32 -04:00
dependabot[bot]
988fc0941b build(deps): bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 (#650)
Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.6.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-version: 1.6.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-11 16:52:10 +00:00
Xe Iaso
f5140ae57b test: introduce SSH based CI for non-native test hosts (#644)
* feat: ssh based CI

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test: implement SSH ci with caches and github actions

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): fix known hosts secret

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): clone the repo, that's important

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): speed up ci by prebaking the SSH CI image

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): set -euo

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): enable pull_request_target so things work

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): oh goody it's broken

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): add cronjob to rebuild ci runner image

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): also run yeet

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): force git version for yeet

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): run set -x in the container

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): fix yeet?

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): remove yeet for now

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): disable for PRs for now

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-11 12:50:01 -04:00
Jason Cameron
bbdee34f37 fix(anubis): improve challenge handling and error reporting (#645) 2025-06-11 12:47:06 -04:00
Stephanie Gawroriski
6e2eeb9e65 Update known-instances.md to include SquirrelJME (#643)
Include SquirrelJME, which is an emulator for Java ME 8.

Signed-off-by: Stephanie Gawroriski <xer@multiphasicapps.net>
2025-06-10 23:20:50 +00:00
Xe Iaso
c638653172 feat(lib): implement request weight (#621)
* feat(lib): implement request weight

Replaces #608

This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): maintain legacy challenge behavior

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): make weight have dedicated checkers for the hashes

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(data): convert some rules over to weight points

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: document request weight

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(CHANGELOG): spelling error

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: fix links to challenge information

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(policies): fix formatting

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(config): make default weight adjustment 5

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-09 15:25:04 -04:00
Fierelier
0fe46b48cf Make progress bar styling more compatible (UXP, etc) (#636)
* Make progress bar styling more compatible (UXP, etc)

* Add 'Make progress bar styling more compatible (UXP, etc)'

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Fierelier <fier@airmail.cc>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-09 12:19:38 -04:00
David Chandek-Stark
d6e5561768 Adds ability to toggle off stripping of private addrs from XFF (#619)
* Adds ability to toggle off stripping of private addrs from XFF

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: refactor to flow better

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-09 13:33:19 +00:00
dependabot[bot]
6594ae0eef build(deps): bump github/codeql-action in the github-actions group (#635)
Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 3.28.18 to 3.28.19
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](ff0a06e83c...fca7ace96b)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 3.28.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-09 08:56:41 -04:00
Lothar Serra Mari
ad09f82c3c docs(admin/environments): Prefer IPv6 over IPv4 for apache2 listener directive (#628)
Signed-off-by: Lothar Serra Mari <mail@serra.me>
2025-06-09 08:56:30 -04:00
Xe Iaso
372b797f64 chore: go generate
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-08 20:52:22 -04:00
dependabot[bot]
6eaf0e13a2 build(deps): bump the gomod group with 2 updates (#634)
Bumps the gomod group with 2 updates: [github.com/a-h/templ](https://github.com/a-h/templ) and [golang.org/x/net](https://github.com/golang/net).


Updates `github.com/a-h/templ` from 0.3.887 to 0.3.898
- [Release notes](https://github.com/a-h/templ/releases)
- [Changelog](https://github.com/a-h/templ/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/a-h/templ/compare/v0.3.887...v0.3.898)

Updates `golang.org/x/net` from 0.40.0 to 0.41.0
- [Commits](https://github.com/golang/net/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.898
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: gomod
- dependency-name: golang.org/x/net
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: gomod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-08 20:44:25 -04:00
Dryusdan
281b6c5c00 Bump ai.robots.txt to v1.34 (#632) 2025-06-08 14:54:47 -04:00
Jason Cameron
9539668049 style: Some minor fixes (#548)
* chore(deps): update dependencies in go.mod and go.sum

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* refactor: rename variables for clarity in anubis.go and main.go

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(checker): handle error when inserting IP range in ranger

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(tests): simplify boolean checks in header and URL value tests

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* refactor(api): remove unused /test-error endpoint and restrict /make-challenge to development

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* build(deps): update golang-set to v2.8.0 in go.sum

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update metadata

check-spelling run (pull_request) for json/stuff

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

---------

Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
2025-06-07 18:21:22 +00:00
Xe Iaso
8eff57fcb6 chore(docs/manifest): try no-js challenge to see how it impacts false positive rate
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-06 21:40:28 -04:00
Xe Iaso
4ac59c3a79 feat(lib/challenge): HTTP meta refresh challenge method (#623)
* feat(lib/challenge): HTTP meta refresh challenge method

Closes #95

This challenge method enables users that don't (or won't) support
JavaScript to pass Anubis challenges. It works by using HTML meta
refresh directives to ensure that the client is a browser.

This is OFF by default. In order to enable it, an administrator MUST
choose to make the default challenge method `metarefresh`.

TODO(Xe):

- [ ] Documentation on this challenge method
- [ ] Amend wording around Anubis being a proof of work proxy in the docs
- [ ] Add configuration file syntax for the default challenge method and settings
- [ ] Test with early customers

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib/challenge/metarefresh): use this value of err

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: add metarefresh challenge info, Web AI Firewall Utility

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-06 21:18:55 -04:00
Lothar Serra Mari
bee1c22b96 docs(known-instances): add wiki.dolphin-emu.org to known instances (#626)
Signed-off-by: Lothar Serra Mari <mail@serra.me>
2025-06-06 13:35:24 -04:00
Xe Iaso
5a7499ea3b fix(lib/challenge): allow challenges to register HTTP routes (#620)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-06 00:26:23 +00:00
Jan Pieter Waagmeester
5f3861ab37 docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth" (#615)
* docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth"

Name definition:

76fa3e01a5/anubis.go (L12-L14)

The name changed in 6c0ff3f4d5


Signed-off-by: Jan Pieter Waagmeester <jieter@jieter.nl>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Jan Pieter Waagmeester <jieter@jieter.nl>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-05 20:59:16 +00:00
foosinn
9f1d791991 docs(subrequest-auth): document required policy changes (#613)
* docs(subrequest-auth): document required policy changes

Signed-off-by: foosinn <foosinn@f2o.io>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: foosinn <foosinn@f2o.io>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-05 16:53:18 -04:00
Markus Sommer
76fa3e01a5 docs(known-instances): add Alliance of Hessian Libraries (#611)
Signed-off-by: Markus Sommer <markus@splork.de>
2025-06-04 02:03:57 +00:00
Xe Iaso
f2db43ad4b feat: implement challenge registry (#607)
* feat: implement challenge method registry

This paves the way for implementing a no-js check method (#95) by making
the challenge providers more generic.

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib/challenge): rename proof-of-work package to proofofwork

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): make validated challenges a CounterVec

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): annotate jwts with challenge method

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib/challenge/proofofwork): implement tests

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib): add smoke tests for known good and known bad config files

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): use challenge.Impl#Issue when issuing challenges

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-04 02:01:58 +00:00
Xe Iaso
ba4412c907 chore(sponsors): add Raptor Computing Systems
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-03 17:49:28 -04:00
Xe Iaso
f184cd81e7 docs(faq): anubis does not mine bitcoin (#609)
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-03 07:14:41 -04:00
Xe Iaso
59bfced8bf docs(admin/environments): update suggested HTTP headers
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-03 06:57:37 -04:00
Xe Iaso
780a935cb8 chore(sponsors): add wildbase
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-03 06:18:40 -04:00
Xe Iaso
f4bc1df797 chore(sponsors): add Uberspace
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-02 09:42:13 -04:00
dependabot[bot]
b496c90e86 build(deps): bump github.com/a-h/templ in the gomod group (#601)
Bumps the gomod group with 1 update: [github.com/a-h/templ](https://github.com/a-h/templ).


Updates `github.com/a-h/templ` from 0.3.865 to 0.3.887
- [Release notes](https://github.com/a-h/templ/releases)
- [Changelog](https://github.com/a-h/templ/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/a-h/templ/compare/v0.3.865...v0.3.887)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.887
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: gomod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-01 23:39:42 -04:00
dependabot[bot]
ec73bcbaf1 build(deps): bump docker/build-push-action in the github-actions group (#602)
Bumps the github-actions group with 1 update: [docker/build-push-action](https://github.com/docker/build-push-action).


Updates `docker/build-push-action` from 6.17.0 to 6.18.0
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](1dc7386353...263435318d)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: 6.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-01 23:39:05 -04:00
dependabot[bot]
8d19eed200 build(deps-dev): bump esbuild from 0.25.4 to 0.25.5 in the npm group (#600)
Bumps the npm group with 1 update: [esbuild](https://github.com/evanw/esbuild).


Updates `esbuild` from 0.25.4 to 0.25.5
- [Release notes](https://github.com/evanw/esbuild/releases)
- [Changelog](https://github.com/evanw/esbuild/blob/main/CHANGELOG.md)
- [Commits](https://github.com/evanw/esbuild/compare/v0.25.4...v0.25.5)

---
updated-dependencies:
- dependency-name: esbuild
  dependency-version: 0.25.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-01 23:38:45 -04:00