Are You Accidentally Blocking AI Crawlers?

A surprising number of businesses block AI crawlers in robots.txt or at the edge without meaning to, then wonder why they are not visible in AI search. The blocks usually come from security plugins, bot-protection defaults, or copied robots.txt files rather than a deliberate decision. The most-affected crawlers tend to be the best known (GPTBot, ClaudeBot). This is a practical how-to-check guide, not a measured survey, so it does not quote a specific block rate.

Which crawlers get blocked

When AI crawlers are blocked, it is most often the widely-named ones. The pattern looks like this:

Why blocks are usually unintentional

Sites that block AI crawlers very often still have active SEO signals: optimised titles, meta descriptions, analytics, and sitemaps. They clearly want to be found online, which suggests the AI-crawler block was not a deliberate choice.

The most common causes:

• WordPress security plugins - Wordfence, Sucuri, and iThemes Security sometimes add blanket bot blocks. These include AI crawlers. The site owner installs the plugin for security and never checks which bots get blocked
• Cloudflare bot protection - Bot Fight Mode and Super Bot Fight Mode can block AI crawlers. This setting is often on by default
• SEO plugins with AI opt-outs - Yoast and Rank Math added AI crawler blocking options. Some are on by default
• Copied robots.txt files - Sites built from templates sometimes inherit rules that block crawlers the original developer added
• Hosting provider defaults - Some managed WordPress hosts add AI crawler blocks at the server level

The GPTBot distinction matters

Some businesses block GPTBot (training data) on purpose while allowing OAI-SearchBot (search indexing). They do not want to train AI models but do want to appear in ChatGPT search results. This is a valid choice.

But our data shows several sites that block GPTBot without knowing OAI-SearchBot exists. They blocked the well-known crawler and left the lesser-known one unaddressed. This gap suggests the blocks were not planned.

Training vs search crawlers: the new distinction

Note: our research came before AI platforms split their crawlers into training and search bots. The smart approach now is to block training bots (GPTBot, ClaudeBot, Google-Extended) while allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking all AI crawlers still hurts you. But blocking training bots while allowing search bots protects your content and keeps you visible.

Business impact

Block OAI-SearchBot and ChatGPT search cannot find you. Block Claude-SearchBot and Claude cannot browse your pages. Block PerplexityBot and you vanish from Perplexity results. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) is a valid choice. It does not affect AI search visibility. It only controls whether your content trains future models.

ChatGPT is the largest AI search platform and AI referral traffic is growing quickly off a small base. Blocking search crawlers means missing a fast-growing channel.

How to check and fix

1. Check your robots.txt - visit yoursite.com/robots.txt and search for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended
2. Check Cloudflare - if you use Cloudflare, go to Security > Bots and review your bot management settings
3. Check WordPress plugins - review security plugin settings for any bot blocking rules
4. Explicitly allow - add User-agent: GPTBot / Allow: / lines for each AI crawler you want to allow

See our complete robots.txt configuration guide for the exact code to copy into your file.

See all AI search statistics · Get a free crawler access audit