23% of UK Business Websites Block AI Crawlers

Our audit found that 7 out of 30 UK SME websites (23%) block at least one AI crawler in their robots.txt. GPTBot is the most commonly blocked at 23.3%, followed by ClaudeBot at 16.7% and PerplexityBot at 3.3%. Most of these blocks appear to be unintentional — the same sites had active SEO signals suggesting they want to be found online.

Blocking rates by crawler

AI Crawler	Platform	Sites blocking	Block rate
GPTBot	OpenAI (ChatGPT training)	7 / 30	23.3%
ClaudeBot	Anthropic (Claude)	5 / 30	16.7%
OAI-SearchBot	OpenAI (ChatGPT search)	4 / 30	13.3%
Google-Extended	Google (Gemini/AI Overviews)	2 / 30	6.7%
PerplexityBot	Perplexity	1 / 30	3.3%
CCBot	Common Crawl	6 / 30	20.0%

Why blocks are usually unintentional

The telling detail in our audit was that nearly all sites blocking AI crawlers also had active SEO signals — optimised title tags, meta descriptions, Google Analytics, submitted sitemaps. These are businesses actively trying to be found online. They're blocking AI crawlers without knowing it.

The most common causes:

• WordPress security plugins — plugins like Wordfence, Sucuri, and iThemes Security sometimes add blanket bot blocks that include AI crawlers. The site owner installs the plugin for security and doesn't check which bots get blocked
• Cloudflare bot protection — Cloudflare's "Bot Fight Mode" and Super Bot Fight Mode can block or challenge AI crawlers. The setting is often enabled by default on business plans
• SEO plugins with AI opt-outs — some SEO plugins (including Yoast and Rank Math) added AI crawler blocking options, sometimes enabled by default, in response to the AI training data debate
• Copied robots.txt files — sites using templates or migrated from other platforms sometimes inherit robots.txt rules that block crawlers the original developer added
• Hosting provider defaults — some managed WordPress hosts add AI crawler blocks at the server level

The GPTBot distinction matters

Some businesses deliberately block GPTBot (training data) while allowing OAI-SearchBot (search indexing), reasoning that they don't want to contribute to AI training but do want to appear in ChatGPT search results. This is a valid strategic choice.

However, our data shows several sites blocking GPTBot without even knowing OAI-SearchBot exists — meaning they've blocked the more well-known crawler and left the less-known one unaddressed in either direction. This inconsistency suggests the blocks weren't strategic decisions.

Business impact

If your site blocks GPTBot and OAI-SearchBot, ChatGPT cannot crawl your content and is significantly less likely to cite you. If you block ClaudeBot, Claude can't access your pages. If you block Google-Extended, you may be excluded from Gemini and AI Overviews training data.

Given that ChatGPT accounts for 78% of AI search market share and AI referral traffic is growing 1,300% year-over-year, blocking these crawlers means missing a rapidly expanding channel.

How to check and fix

1. Check your robots.txt — visit yoursite.com/robots.txt and search for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended
2. Check Cloudflare — if you use Cloudflare, go to Security > Bots and review your bot management settings
3. Check WordPress plugins — review security plugin settings for any bot blocking rules
4. Explicitly allow — add User-agent: GPTBot / Allow: / lines for each AI crawler you want to allow

See our complete robots.txt configuration guide for the exact code to copy into your file.

See all AI search statistics · Get a free crawler access audit