What Should My robots.txt Look Like for AI Search?
Your robots.txt needs to explicitly allow AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. Here's the exact configuration to copy.
Your robots.txt should explicitly allow all major AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and GoogleOther. 23% of UK business websites block at least one of these — often without knowing it. If AI crawlers can't access your site, AI platforms can't cite you.
The recommended robots.txt configuration
Copy this and add it to your robots.txt file:
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: GoogleOther
Allow: /
User-agent: CCBot
Allow: /What each crawler does
| Crawler | Platform | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | Real-time search indexing for ChatGPT search results (NOT training) |
| ChatGPT-User | OpenAI | Custom GPT requests — when users ask ChatGPT to visit your site |
| ClaudeBot | Anthropic | Claude's web browsing and training |
| PerplexityBot | Perplexity | Real-time source indexing for Perplexity answers |
| Google-Extended | Gemini and AI Overviews training | |
| GoogleOther | Additional Google AI crawling | |
| CCBot | Common Crawl | Open dataset used by many AI models |
How to check your current robots.txt
Go to yoursite.com/robots.txt in a browser. If you see Disallow: / for any of the crawlers above, that platform is blocked from accessing your content.
Common mistakes
- WordPress default blocks — some WordPress themes and security plugins add AI crawler blocks by default
- Cloudflare bot protection — Cloudflare's bot management can inadvertently block AI crawlers. Check your Cloudflare settings under Security > Bots
- Blocking GPTBot but allowing OAI-SearchBot — you need both for full ChatGPT visibility
- Not having a robots.txt at all — this is actually fine (all bots are allowed by default) but explicitly allowing is better practice
Related questions
Oliver Mackman
AI Search Analyst, SEOCompare
Oliver leads SEOCompare's editorial and comparison research. With over a decade in digital marketing, he oversees agency evaluation, tool testing, and AI search data analysis.
Last reviewed: 7 April 2026
Need help with AI search visibility?
Get a free AI visibility audit to see how your business appears across ChatGPT, Gemini, Perplexity, and AI Overviews.
Request your free audit