Robots.txt checker for AI crawlers
See which AI bots your robots.txt lets in: ChatGPT, Claude, Perplexity, Gemini and Google. Bot by bot, with the lines to fix it. Free, no sign-up.
AI bots
acme.com/robots.txt
The bots can read your site. Do they recommend you?
Letting AI bots in is the first step. It doesn't tell you what ChatGPT, Claude, Gemini and Perplexity actually say to buyers, who they name instead of you, or what they get wrong.
Check my product $29The AI bots that matter
Each AI company uses more than one bot. Some find pages for answers people see today; others collect pages for training. You can allow one kind and block the other.
| Bot | Company | What it does |
|---|---|---|
OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers |
ChatGPT-User | OpenAI | Opens a page when a ChatGPT user asks about it OpenAI says robots.txt rules may not apply to it, because a person asked. |
Claude-SearchBot | Anthropic | Finds pages for Claude search answers |
Claude-User | Anthropic | Opens a page when a Claude user asks about it |
PerplexityBot | Perplexity | Finds and links pages in Perplexity answers |
Perplexity-User | Perplexity | Opens a page when a Perplexity user asks about it Perplexity says it generally ignores robots.txt, because a person asked. |
Googlebot | Google Search, including AI Overviews | |
Google-Extended | Gemini: training, and answers grounded in Google Search Not a separate crawler: a setting Google reads. It does not affect Google Search. | |
GPTBot | OpenAI | Collects pages that may train OpenAI models |
ClaudeBot | Anthropic | Collects pages that may train Claude models |
To let AI answers use your site, block training only
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: * Allow: /
Bot names and roles from each company's own documentation: OpenAI, Anthropic, Perplexity, Google. Checked November 2026.
Questions
Does blocking GPTBot remove me from ChatGPT?
No. OpenAI uses GPTBot to collect pages for training and OAI-SearchBot to find pages for ChatGPT search answers. You can block GPTBot and still allow OAI-SearchBot. The same split exists at Anthropic: ClaudeBot for training, Claude-SearchBot for search.
Why do some bots ignore robots.txt?
ChatGPT-User, Claude-User and Perplexity-User open a page because a person asked about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. They are shown here so you know what your file says to them.
Does Google-Extended affect AI Overviews?
Google says Google-Extended does not affect Google Search. It controls whether Gemini may train on your pages and use them to ground answers in the Gemini app. AI Overviews are part of Google Search, which uses Googlebot.
robots.txt allows the bots. Can they still be blocked?
Yes. This check reads robots.txt only. A firewall or CDN setting can turn bots away before they reach your pages. Cloudflare, for example, has a setting that blocks AI bots. If AI assistants never seem to read your site, check those settings too.
Where is robots.txt?
At the root of your domain: https://yourdomain.com/robots.txt, as plain text. Each subdomain has its own. If there is no file, every bot may read everything.
Also free: the AI readiness checker and the llms.txt checker and generator.