Remove OpenAI bots from UA sniffing

[kennethkalmer]

Summary

• Remove OpenAI's ChatGPT-User and GPTBot from User-Agent sniffing in nginx config
• Update tests to reflect removal (23 → 21 tests)
• OpenAI bots will now receive HTML by default unless they use proper Accept headers or request .md URLs directly

Context

We likely tripped OpenAI's stricter fetch consistency checks because we historically served different content based on UA (even though we also support proper content negotiation). OpenAI compares responses across multiple fetch modes (direct, cached, headless), and UA-based divergence can cause certain URLs to be excluded from their sandbox fetcher — even though we still see ChatGPT traffic elsewhere. Excluding OpenAI bots from UA sniffing (and letting Accept headers / explicit .md URLs decide) should resolve this over time; this also explains why we don't see the same issue with Claude or others.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests

  • Reduced test suite from 23 to 20 tests with corresponding assertion updates.

• Chores

  • Updated LLM bot detection configuration.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The PR removes support for detecting two specific LLM bot User-Agents (ChatGPT-User and GPTBot) by deleting their regex patterns from Nginx bot-detection configuration and removing corresponding test cases, reducing the test suite from 23 to 20 assertions.

Changes

Cohort / File(s)	Summary
Bot Detection Configuration config/nginx.conf.erb	Removed two user-agent regex patterns ("\*ChatGPT-User" and "\*GPTBot") from the is_llm_bot mapping
Test Suite bin/assert-content-negotiation.sh	Removed three test invocations for bot User-Agent handling (ChatGPT-User and GPTBot) and updated expected test count from 23 to 20

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

Two bots bid farewell to our nginx gate,
ChatGPT-User and GPTBot, no longer in fate,
Twenty tests remain, streamlined and bright,
The detection logic simplified just right! 🐰✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main changes: removal of OpenAI bot detection from User-Agent sniffing in nginx and corresponding test updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[kennethkalmer]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@bin/assert-content-negotiation.sh`:
- Line 155: The final echo message currently hardcodes "✅ All 20 tests passed!"
which is incorrect because there are 21 run_test invocations; update the echo in
bin/assert-content-negotiation.sh (the line with echo "✅ All 20 tests passed!")
to reflect the correct count (e.g., "✅ All 21 tests passed!") or, better,
compute the total dynamically from the run_test calls and print that number so
the message always matches the actual number of tests.