robots.txt for AI crawlers: which bots to allow, and which to think about
The user agents behind ChatGPT, Claude, Perplexity and Gemini, what each one does, a robots.txt you can copy, and the firewall setting that overrides it.
Each AI company runs more than one crawler, and they do different jobs. Some fetch pages so the assistant can cite them in live answers. Others collect training data for future models. Blocking the first kind removes you from today's answers. Blocking the second only affects what future models learn. Treating them as one group is the most common robots.txt mistake we see.
The crawlers that matter
| Company | User agent | What it does |
|---|---|---|
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search results |
| OpenAI | ChatGPT-User | Fetches a page when a user's conversation needs it |
| OpenAI | GPTBot | Collects training data |
| Anthropic | Claude-SearchBot | Indexes pages for Claude's search |
| Anthropic | Claude-User | Fetches a page when a user asks Claude about it |
| Anthropic | ClaudeBot | Collects training data |
| Perplexity | PerplexityBot | Indexes pages for Perplexity answers |
| Perplexity | Perplexity-User | Fetches pages in response to a user's question |
Googlebot | Crawls for Google Search, including AI Overviews | |
Google-Extended | Controls use of your content for Gemini models; does not affect Search | |
| Microsoft | Bingbot | Crawls for Bing, which Copilot and other assistants search |
Vendors add and rename agents, so check their crawler documentation once or twice a year.
A robots.txt you can start from
This allows search engines and every AI crawler, keeps them out of private paths, and points them to your sitemap:
User-agent: *
Allow: /
Disallow: /account/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlA User-agent: * group already covers every crawler that has no group of its own. Problems start when a site has an older block such as User-agent: GPTBot followed by Disallow: /, often added in 2023 when blocking AI training was a popular default. That rule still applies today.
If you decide to opt out of training while staying in live answers, block only the training agents:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /Leave OAI-SearchBot, Claude-SearchBot, PerplexityBot, the -User fetchers, Googlebot and Bingbot allowed.
Training crawlers: a real trade-off
Blocking training crawlers keeps your content out of future model versions. The cost is that those models will know less about your brand when they answer without searching. For a publisher selling original content, blocking can be the right call. For a software company that wants to be recommended, allowing them usually serves you better.
The firewall can override robots.txt
robots.txt is a request. A CDN or firewall can refuse the crawler before it ever reads the file. Cloudflare offers a one-click setting to block AI crawlers, and has made it the default option for some new domains. Bot protection on Vercel, Akamai and other hosts can have the same effect.
Check from the outside by requesting a page with a crawler's user agent:
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
https://yourdomain.com/A 403, a 429 or a challenge page means real crawlers are probably turned away too. The free AI readiness audit runs this test for four AI crawlers and reads your robots.txt rules for eleven.
FAQ
Do I need a separate rule for every AI crawler?
No. If your User-agent: * group allows crawling, every crawler without its own group is allowed. Separate groups only matter when you want different rules for a specific agent.
Does blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended does not affect Google Search, and AI Overviews are part of Search. Googlebot governs both.
What about llms.txt?
llms.txt is a proposed file that summarizes your site for language models. It costs little to add, but the major assistants have not committed to reading it, so treat it as optional and get robots.txt and your firewall right first.