AI Crawler User-Agent List
Every major AI crawler, what it does, and a robots.txt snippet to allow or block it.
Crawlers
| User agent | Operator | Purpose | Respects robots.txt | What it does | Docs | Your choice |
|---|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes | Collects pages to train OpenAI's models. | Docs (opens in a new tab) | |
| ChatGPT-User | OpenAI | User fetch | Yes | Fetches a page when a ChatGPT user asks about it during a conversation. | Docs (opens in a new tab) | |
| OAI-SearchBot | OpenAI | Search | Yes | Indexes pages so they can appear in ChatGPT search results. | Docs (opens in a new tab) | |
| ClaudeBot | Anthropic | Training | Yes | Collects pages to train Anthropic's Claude models. | Docs (opens in a new tab) | |
| anthropic-ai | Anthropic | Training | Yes | Legacy Anthropic training token. Many robots.txt files still list it. | Docs (opens in a new tab) | |
| Claude-User | Anthropic | User fetch | Yes | Fetches a page when a Claude user asks about it during a conversation. | Docs (opens in a new tab) | |
| Claude-SearchBot | Anthropic | Search | Yes | Indexes pages to improve the results Claude cites. | Docs (opens in a new tab) | |
| Google-Extended | Training | Yes | Controls whether your content trains Gemini and appears in grounding. Googlebot is separate. | Docs (opens in a new tab) | ||
| PerplexityBot | Perplexity | Search | Reports mixed | Indexes pages for Perplexity's search index. | Docs (opens in a new tab) | |
| Perplexity-User | Perplexity | User fetch | Reports mixed | Fetches a page when a Perplexity user asks about it. | Docs (opens in a new tab) | |
| Bytespider | ByteDance | Training | Reports mixed | ByteDance's crawler, used for training and for TikTok search. | Docs (opens in a new tab) | |
| CCBot | Common Crawl | Training | Yes | Builds the open Common Crawl archive that many models train on. | Docs (opens in a new tab) | |
| Applebot-Extended | Apple | Training | Yes | Controls whether Applebot's crawl trains Apple's models. Applebot itself is separate. | Docs (opens in a new tab) | |
| Amazonbot | Amazon | Mixed | Yes | Crawls for Alexa answers and Amazon's models. | Docs (opens in a new tab) | |
| Meta-ExternalAgent | Meta | Training | Yes | Collects pages to train Meta's models. | Docs (opens in a new tab) | |
| DuckAssistBot | DuckDuckGo | Search | Yes | Fetches sources for DuckDuckGo's AI answers. | Docs (opens in a new tab) | |
| cohere-ai | Cohere | Training | Yes | Collects pages for Cohere's models. | Docs (opens in a new tab) | |
| Diffbot | Diffbot | Mixed | Yes | Extracts structured data for Diffbot's knowledge graph and its customers. | Docs (opens in a new tab) | |
| YouBot | You.com | Search | Yes | Indexes pages for You.com's AI search. | Docs (opens in a new tab) | |
| MistralAI-User | Mistral | User fetch | Yes | Fetches a page when a Le Chat user asks about it. | Docs (opens in a new tab) |
robots.txt snippet
Pick Block or Allow on any row and the snippet shows up here.
How it works
- 1
Browse the list or filter by operator and purpose.
- 2
Pick the crawlers you want to allow or block.
- 3
Copy the generated robots.txt snippet.
Questions
Should I block AI crawlers?
Blocking training crawlers keeps your content out of future models. Blocking search and user-fetch crawlers removes you from AI answers. Most brands that want AI visibility allow the search and user agents and decide on training separately.
Do all of these crawlers respect robots.txt?
OpenAI, Anthropic, Google, Apple and Common Crawl publish that they do. Reports on Bytespider and some Perplexity traffic are mixed. robots.txt is a request, not a wall.
Related tools
Track this across ChatGPT, Claude and Gemini every day
BrandVisibility.ai runs your queries on every engine and reports mentions, citations and share of voice. Free trial, no card.