Files
anubis-mirror/data/crawlers/openai-searchbot.yaml
Xe Iaso bf5d66222c chore: set up commitlint, husky, and prettier (#1451)
* chore: add prettier configuration

Signed-off-by: Xe Iaso <me@xeiaso.net>

* format: run prettier tree-wide

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(prettier): ignore intentionally ungrammatical files

Signed-off-by: Xe Iaso <me@xeiaso.net>

* ci: add PR title lint rule

Signed-off-by: Xe Iaso <me@xeiaso.net>

* ci: add DCO check

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: add commitlint and husky

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: add CONTRIBUTING guidelines

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: set SKIP_INTEGRATION in precommit tests

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: update spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* ci(dco): remove reopened trigger

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: remove dead file

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(prettier): don't format nginx includes

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2026-02-15 08:19:12 -05:00

15 lines
435 B
YAML

# Indexing for search, does not collect training data
# https://platform.openai.com/docs/bots/overview-of-openai-crawlers
- name: openai-searchbot
user_agent_regex: OAI-SearchBot/1\.0; \+https\://openai\.com/searchbot
action: ALLOW
# https://openai.com/searchbot.json
remote_addresses:
[
"20.42.10.176/28",
"172.203.190.128/28",
"104.210.140.128/28",
"51.8.102.0/24",
"135.234.64.0/24",
]