Two separate decisions
OpenAI documents OAI-SearchBot as a search-discovery crawler and GPTBot as a crawler associated with potential model training. Disallowing GPTBot doesn't inherently require blocking OAI-SearchBot. Anthropic similarly distinguishes Claude-SearchBot and ClaudeBot.
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Disallow: /
This is only an illustration of one publisher's preference — not a universal recommendation. Review policies carefully if you rely on AI search traffic or have content licensing requirements.
What robots.txt can reveal
For a given user agent and page path, the Robots Exclusion Protocol selects the most specific matching allow/disallow rule. Important: robots.txt is not a security boundary. It cannot protect secrets or enforce licensing against malicious crawlers. Use authentication to protect non-public content.
What it cannot reveal
Allowing a bot does not prove that your website has been crawled, indexed, retrieved, cited or recommended. Google says its AI Overview and AI Mode features rely on the usual Search best practices; no special AI-only markup is required. Search Console and your server logs are stronger sources for measurable website activity.
An important separation: robots.txt describes requested crawler behavior. Firewalls, CDN rules, HTTP errors and actual indexing decisions can still stop access.
Try a controlled test
Open the AI crawler debugger, paste the site's actual robots.txt and enter the URL path you want to check. Compare search and training agents separately.