* feat: first implementation of honeypot logic
This is a bit of an experiment, stick with me.
The core idea here is that badly written crawlers are that: badly
written. They look for anything that contains `<a href="whatever" />`
tags and will blindly use those values to recurse. This takes advantage
of that by hiding a link in a `<script>` tag like this:
```html
<script type="ignore"><a href="/bots-only">Don't click</a></script>
```
Browsers will ignore it because they have no handler for the "ignore"
script type.
This current draft is very unoptimized (it takes like 7 seconds to
generate a page on my tower), however switching spintax libraries will
make this much faster.
The hope is to make this pluggable with WebAssembly such that we force
administrators to choose a storage method. First we crawl before we
walk.
The AI involvement in this commit is limited to the spintax in
affirmations.txt, spintext.txt, and titles.txt. This generates a bunch
of "pseudoprofound bullshit" like the following:
> This Restoration to Balance & Alignment
>
> There's a moment when creators are being called to realize that the work
> can't be reduced to results, but about energy. We don't innovate products
> by pushing harder, we do it by holding the vision. Because momentum can't
> be forced, it unfolds over time when culture are moving in the same
> direction. We're being invited into a paradigm shift in how we think
> about innovation. [...]
This is intended to "look" like normal article text. As this is a first
draft, this sucks and will be improved upon.
Assisted-by: GLM 4.6, ChatGPT, GPT-OSS 120b
Signed-off-by: Xe Iaso <[email protected]>
* fix(honeypot/naive): optimize hilariously
Signed-off-by: Xe Iaso <[email protected]>
* feat(honeypot/naive): attempt to automatically filter out based on crawling
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): use mazeGen instead of bsGen
Signed-off-by: Xe Iaso <[email protected]>
* docs: add honeypot docs
Signed-off-by: Xe Iaso <[email protected]>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <[email protected]>
* chore: fix spelling metadata
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
This is not used yet, but it will be part of a larger strategy around
adding/removing weight based on JA4H (and other) fingerprint matches
with Thoth.
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/policy/expressions): add system load average to bot expression inputs
This lets Anubis dynamically react to system load in order to
increase and decrease the required level of scrutiny. High load? More
scrutiny required. Low load? Less scrutiny required.
* docs: spell system correctly
Signed-off-by: Xe Iaso <[email protected]>
* Update metadata
check-spelling run (pull_request) for Xe/load-average
Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>
* fix(default-config): don't enable low load average feature by default
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: check-spelling-bot <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
* feat(decaymap): add Delete method
Signed-off-by: Xe Iaso <[email protected]>
* chore(lib/challenge): refactor Validate to take ValidateInput
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib): implement store interface
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): all metapackage to import all store implementations
Signed-off-by: Xe Iaso <[email protected]>
* chore(policy): import all store backends
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib): use new challenge creation flow
Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.
This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.
Signed-off-by: Xe Iaso <[email protected]>
* chore(decaymap): fix documentation typo
Signed-off-by: Xe Iaso <[email protected]>
* chore(lib): fix SA4004
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/store): make generic storage interface test adaptor
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* fix(decaymap): invert locking process for Delete
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): add bbolt store implementation
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* chore: go mod tidy
Signed-off-by: Xe Iaso <[email protected]>
* chore(devcontainer): adapt to docker compose, add valkey service
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): make challenges live for 30 minutes by default
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): implement valkey backend
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/store/valkey): disable tests if not using docker
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/policy/config): ensure valkey stores can be loaded
Signed-off-by: Xe Iaso <[email protected]>
* Update metadata
check-spelling run (pull_request) for Xe/store-interface
Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>
* chore(devcontainer): remove port forwards because vs code handles that for you
Signed-off-by: Xe Iaso <[email protected]>
* docs(default-config): add a nudge to the storage backends section of the docs
Signed-off-by: Xe Iaso <[email protected]>
* chore(docs): listen on 0.0.0.0 for dev container support
Signed-off-by: Xe Iaso <[email protected]>
* docs(policy): document storage backends
Signed-off-by: Xe Iaso <[email protected]>
* docs: update CHANGELOG and internal links
Signed-off-by: Xe Iaso <[email protected]>
* docs(admin/policies): don't start a sentence with as
Signed-off-by: Xe Iaso <[email protected]>
* chore: fixes found in review
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: check-spelling-bot <[email protected]>
* lib/localization: implement localization system
Locale files are placed in lib/localization/locales/. If you add a
locale, update manifest.json with available locales.
* Exclude locales from check spelling
* tests(lib/localization): add comprehensive translations test
Signed-off-by: Xe Iaso <[email protected]>
* fix(challenge/metarefresh): enable localization
Signed-off-by: Xe Iaso <[email protected]>
* fix: use simple syntax for localization in templ
Also localize CELPHASE into French according to the wishes of the
artist.
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* chore:(js): fix forbidden patterns
Signed-off-by: Xe Iaso <[email protected]>
* chore: add goi18n to tools
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/localization): dynamically determine the list of supported languages
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
* chore(xess): remove unused xess templates
Signed-off-by: Jason Cameron <[email protected]>
* chore(checker): remove unused staticHashChecker implementation
Signed-off-by: Jason Cameron <[email protected]>
* feat: add pinact and deadcode to go tools (pinact is used for the gha pinning)
Signed-off-by: Jason Cameron <[email protected]>
* chore: update Docker and kubectl actions to latest versions
Signed-off-by: Jason Cameron <[email protected]>
* chore: update Homebrew action from master to main in workflow files
See https://github.com/Homebrew/actions/commit/df537ec97fb77a615dc06a860b3d5b6d9c256cdb
Signed-off-by: Jason Cameron <[email protected]>
* chore: remove unused go-colorable and tools dependencies from go.sum
Signed-off-by: Jason Cameron <[email protected]>
* chore: update postcss-import and other dependencies to latest versions
Signed-off-by: Jason Cameron <[email protected]>
* chore: update Docusaurus dependencies to version 3.8.1
Signed-off-by: Jason Cameron <[email protected]>
* chore: downgrade playwright and playwright-core to version 1.52.0
Signed-off-by: Jason Cameron <[email protected]>
---------
Signed-off-by: Jason Cameron <[email protected]>
* feat: replace cidranger with bart improving performance by 3-20x
Signed-off-by: Jason Cameron <[email protected]>
* perf: replace cidranger with bart for IP range checking
- Replace cidranger.Ranger with bart.Lite in RemoteAddrChecker
- Use netip.ParsePrefix instead of net.ParseCIDR for modern IP handling
- Improve performance: 3-20x faster lookups with zero heap allocations
- Update imports to use github.com/gaissmai/bart and net/netip
- Remove cidranger dependency from go.mod
Benchmark results:
- IPv4 lookups: 4x faster (15.58ns vs 63.25ns, 0 vs 2 allocs)
- IPv6 lookups: 3x faster (26.51ns vs 76.96ns, 0 vs 2 allocs)
- Insertions: 20x faster (976ns vs 19,191ns)
- Large tables: 14x faster (5.2ns vs 74.85ns)
* docs: clarify CHANGELOG to not give false impressions
Signed-off-by: Jason Cameron <[email protected]>
* perf: optimize string concatenation in RemoteAddrChecker hash generation
Replace fmt.Fprintln with strings.Join for 7x faster performance:
- Before: 935.1 ns/op, 784 B/op, 22 allocs/op
- After: 133.2 ns/op, 192 B/op, 1 alloc/op
The hash is used for JWT cookie validation and error code generation.
Comma separation provides the same deterministic uniqueness as newlines
but with significantly better performance during policy initialization.
* chore: remove accidentally commited string benchmark
Signed-off-by: Jason Cameron <[email protected]>
* style: apply Copilot suggestions
Signed-off-by: Jason Cameron <[email protected]>
* fix: reference the right var name
i cannot write a merge commit
Signed-off-by: Jason Cameron <[email protected]>
---------
Signed-off-by: Jason Cameron <[email protected]>
* chore(deps): update dependencies in go.mod and go.sum
Signed-off-by: Jason Cameron <[email protected]>
* refactor: rename variables for clarity in anubis.go and main.go
Signed-off-by: Jason Cameron <[email protected]>
* fix(checker): handle error when inserting IP range in ranger
Signed-off-by: Jason Cameron <[email protected]>
* fix(tests): simplify boolean checks in header and URL value tests
Signed-off-by: Jason Cameron <[email protected]>
* refactor(api): remove unused /test-error endpoint and restrict /make-challenge to development
Signed-off-by: Jason Cameron <[email protected]>
* build(deps): update golang-set to v2.8.0 in go.sum
Signed-off-by: Jason Cameron <[email protected]>
* Update metadata
check-spelling run (pull_request) for json/stuff
Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>
---------
Signed-off-by: Jason Cameron <[email protected]>
Signed-off-by: check-spelling-bot <[email protected]>
Gives us many nice things like:
* Windows support for yeet (modulo TecharoHQ/yeet#29)
* Removes the dependency on /bin/sh or /bin/bash thanks to
mvdan.cc/sh/v3
* Checksum-compliant reproducible builds by default
Signed-off-by: Xe Iaso <[email protected]>
This means that yeet's version will be managed by `go.mod` and
auto-bumped with dependabot. This removes human error from the equation
and ensures that Anubis is always built with the newest version of yeet.
This also makes it trivial to make your own local packages for testing:
```text
go tool yeet
```
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/policy): add support for CEL checkers
This adds the ability for administrators to use Common Expression
Language[0] (CEL) for more advanced check logic than Anubis previously
offered.
These can be as simple as:
```yaml
- name: allow-api-routes
action: ALLOW
expression:
and:
- '!(method == "HEAD" || method == "GET")'
- path.startsWith("/api/")
```
or get as complicated as:
```yaml
- name: allow-git-clients
action: ALLOW
expression:
and:
- userAgent.startsWith("git/") || userAgent.contains("libgit") || userAgent.startsWith("go-git") || userAgent.startsWith("JGit/") || userAgent.startsWith("JGit-")
- >
"Git-Protocol" in headers && headers["Git-Protocol"] == "version=2"
```
Internally these are compiled and evaluated with cel-go[1]. This also
leaves room for extensibility should that be desired in the future. This
will intersect with #338 and eventually intersect with TLS fingerprints
as in #337.
[0]: https://cel.dev/
[1]: https://github.com/google/cel-go
Signed-off-by: Xe Iaso <[email protected]>
* feat(data/apps): add API route allow rule for non-HEAD/GET
Signed-off-by: Xe Iaso <[email protected]>
* docs: document expression syntax
Signed-off-by: Xe Iaso <[email protected]>
* fix: fixes in review
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
* feat: Add Open Graph tag support (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* Fix: Prevent nil pointer dereference in test (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* feat!: Implement Open Graph tag caching and passthrough functionality (WIP)
I'm going to sleep. currently tags are passed to renderIndex.
see https://github.com/TecharoHQ/anubis/issues/131
Signed-off-by: Jason Cameron <[email protected]>
* feat: Add configuration for air tool with build and logger settings
Signed-off-by: Jason Cameron <[email protected]>
* feat: Move OG tags to base template (og-tags)
Moves the Open Graph (OG) tags from the index template to
the base template. This allows OG tags to be set on any
page, not just the index. Also adds a
BaseWithOGTags function to the web package to allow
passing OG tags to the base template. Removes the
ogTags parameter from the Index function and template.
Signed-off-by: Jason Cameron <[email protected]>
* Delete CHANGELOG.md
Signed-off-by: Jason Cameron <[email protected]>
* feat: Add language attribute to HTML tag in template
Signed-off-by: Jason Cameron <[email protected]>
* fix(tests): Fix nil pointer ref
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): Add timeout to http client (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* style: fix line endings & indentation
Signed-off-by: Jason Cameron <[email protected]>
* style: add inspection comment for GoBoolExpressions in UnchangingCache
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): Implement Open Graph tag fetching and caching
Signed-off-by: Jason Cameron <[email protected]>
* fix(og-tags): Simplify Open Graph tag extraction logic
Signed-off-by: Jason Cameron <[email protected]>
* fix(og-tags): Add nil check in isOGMetaTag and enhance test cases
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): Add approved tags and prefixes for Open Graph extraction
Signed-off-by: Jason Cameron <[email protected]>
* test(og-tags): Update tests with approved tags and improve clarity
Signed-off-by: Jason Cameron <[email protected]>
* chore: Add changelog notes
Signed-off-by: Jason Cameron <[email protected]>
* fix: Improve stability of the target fetcher?
Signed-off-by: Jason Cameron <[email protected]>
* fix: Update template error handling and improve Open Graph tag integration
Signed-off-by: Jason Cameron <[email protected]>
* style: format files and remove deubg logs
Signed-off-by: Jason Cameron <[email protected]>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* feat: Allow twitter prefixed OG tags by default
Signed-off-by: Jason Cameron <[email protected]>
* chore: replace /tmp with /var
Signed-off-by: Jason Cameron <[email protected]>
* Update docs/docs/CHANGELOG.md
Co-authored-by: Xe Iaso <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
* chore: add fediverse to default prefixes (#og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): Remove og-query-distinct flag
This commit removes the `og-query-distinct` flag and
associated logic. URLs with different query parameters
will now always be treated as the same cache key for Open
Graph tags. This simplifies the caching logic and
improves performance.
Additionally, the http client used for fetching OG tags
is now a member of the OGTagCache struct, rather than a
global variable. This improves testability and allows
for more flexible configuration in the future.
Signed-off-by: Jason Cameron <[email protected]>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
* docs: remove og tags references
Signed-off-by: Jason Cameron <[email protected]>
* refactor: rename url > u to not overlap package name
Signed-off-by: Jason Cameron <[email protected]>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
* fix(tests): Don't use network when network access is disabled
Signed-off-by: Jason Cameron <[email protected]>
* Fix: Handle nil URL in GetOGTags (og-tags)
Signed-off-by: Jason Cameron <[email protected]>
* chore: sort installation docs alphabetically
Signed-off-by: Jason Cameron <[email protected]>
* fix(tests): validate that no duplicate requests are made
Signed-off-by: Jason Cameron <[email protected]>
* style(tests): remove unused ok var
Signed-off-by: Jason Cameron <[email protected]>
* docs: convert to table fmt
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): Enhance OG tag fetching and caching
Adds additional approved OG tags (`keywords`, `author`), improves
Signed-off-by: Jason Cameron <[email protected]>
* chore: update generated templ's after format
Signed-off-by: Jason Cameron <[email protected]>
* fix(tests): update integration_test.go to reflect the new behavior of fetchHTMLDocument
Signed-off-by: Jason Cameron <[email protected]>
* Revert "data/botPolicies: allow iMessage scraper by default (#178)"
This reverts commit 21a9d777
Signed-off-by: Jason Cameron <[email protected]>
* Fix: Simplify ogTags access in cache test.
Didn't know this was possible! wow!
Signed-off-by: Jason Cameron <[email protected]>
* Fix: Handle request timeouts when fetching OG tags (#og-tags)
Cache a nil result for half the TTL to avoid repeatedly
requesting a timed-out URL.
Signed-off-by: Jason Cameron <[email protected]>
* Fix: make OG tags passthrough option function.
Signed-off-by: Jason Cameron <[email protected]>
* Fix: Handle timeouts and non-200 responses when fetching OG tags (og-tags)
- Cache empty results for timeouts and non-200 status codes
to avoid spamming the server.
- Use a non-nil empty map to represent empty results in the
cache, as nil would be a cache miss.
Signed-off-by: Jason Cameron <[email protected]>
* feat(og-tags): switch to http.MaxBytesReader
Signed-off-by: Jason Cameron <[email protected]>
* chore(og-tags): add noindex, nofollow meta tag and update error line numbers
Signed-off-by: Jason Cameron <[email protected]>
---------
Signed-off-by: Jason Cameron <[email protected]>
Signed-off-by: Jason Cameron <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
