robots.txt AI Crawler Checker
See which AI crawlers your robots.txt allows or blocks, by engine.
robots.txt
How it works
- 1
Enter a domain or paste a robots.txt file.
- 2
The checker parses every User-agent group and tests each known AI crawler against its rules, falling back to the wildcard group.
- 3
You get a table per crawler: allowed, blocked, or partially blocked, with the matching rule.
Questions
Which crawlers does it check?
GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, PerplexityBot, Bytespider, CCBot and Applebot-Extended.
What's the difference between GPTBot and ChatGPT-User?
GPTBot collects training data. ChatGPT-User fetches pages on behalf of a user during a conversation. OAI-SearchBot indexes for ChatGPT search. Blocking one doesn't block the others.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content trains Gemini and appears in grounding. Googlebot and search rankings are separate.
Related tools
Track this across ChatGPT, Claude and Gemini every day
BrandVisibility.ai runs your queries on every engine and reports mentions, citations and share of voice. Free trial, no card.