Open your server logs and you'll find visitors with names like GPTBot, ClaudeBot and PerplexityBot. They're crawlers run by AI companies, and whether you let them in is now a real business decision. Block too much and you can disappear from AI-powered search. Allow everything and your content may be used to train AI models.
The good news: most AI companies now run separate crawlers for search and for training, so you can make a nuanced choice.
The Main AI Crawlers and What They Do
| Crawler | Company | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | Finding pages to show in ChatGPT search |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to |
| GPTBot | OpenAI | Collecting content for model training |
| Claude-SearchBot / Claude-User | Anthropic | Search results and user-requested fetches in Claude |
| ClaudeBot | Anthropic | Collecting content for model training |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Google-Extended | Controls use of your content for Gemini — not Google Search | |
| Applebot-Extended | Apple | Controls use of your content for Apple's AI models |
| CCBot | Common Crawl | An open web archive used to train many AI models |
Each company documents its crawlers — for example OpenAI's bot documentation and Google's list of crawlers. Names and behaviour change, so check the source before relying on a rule.
An Important Detail About Google
Blocking Google-Extended does not remove you from Google Search, and it doesn't stop Googlebot crawling your site. Google's AI Overviews in Search are built from Google's normal search index. If you block Googlebot itself, you disappear from Google entirely — almost never what a business wants.
Three Sensible Policies
1. Open (most small businesses)
Allow everything. Your goal is to be found, and your content is marketing anyway. This is effectively the default if your robots.txt doesn't mention AI bots.
2. Search yes, training no (publishers and content-heavy sites)
Allow the search and user-fetch crawlers so you can appear and be cited in AI answers, but block the training crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /3. Closed (rarely right)
Block every AI crawler. This protects content but removes you from a fast-growing discovery channel. It makes sense for paywalled or proprietary content — rarely for a business that wants customers.
The Accidental Block
The most common problem we find isn't a deliberate policy — it's an accidental one. Security plugins, CDN "bot fight" modes and copy-pasted robots.txt templates often block AI crawlers without anyone noticing. Check yours with our free AI Readability Checker, which reads your robots.txt and lists which crawlers are allowed.
What robots.txt Can't Do
robots.txt is a request, not a lock. Reputable crawlers respect it, but it doesn't stop someone copying your content by other means. It also only affects future crawling — it doesn't remove content already collected.
Not sure which policy fits your business? Our AI Readiness Audit reviews your crawler settings, firewall and content — or read what AI SEO is for the bigger picture.