The short answer
- AI companies use separate crawlers for training their models, for building search results and for opening a page when a user asks.
- You can refuse the training crawlers and still allow the search crawlers. They are different names in robots.txt.
- To appear in ChatGPT search, OpenAI says to allow OAI-SearchBot. Perplexity says the same of PerplexityBot.
- robots.txt is a request, not a lock. Some user-requested fetchers do not follow it, and a firewall can refuse a crawler that robots.txt allows.
Three jobs, three kinds of crawler
A crawler is a program that requests pages from your server. AI companies run them for three different jobs.
- Training. The crawler collects pages that may be used to train future models.
- Search. The crawler builds the index that an assistant searches when it answers with links.
- A user’s request. Someone asks the assistant about a page, and it fetches that page there and then.
Each job has its own name. That lets you say yes to one and no to another.
The crawlers by name
These are the names each company publishes in its own documentation.
| Name | Company | What it is for |
|---|---|---|
| GPTBot | OpenAI | Training. Disallowing it tells OpenAI not to use the site’s content to train its models. |
| OAI-SearchBot | OpenAI | Search. It is used to show websites in ChatGPT’s search results. |
| ChatGPT-User | OpenAI | A user’s request. It fetches a page when a ChatGPT user’s action calls for it. |
| OAI-AdsBot | OpenAI | Checks pages that were submitted as ads on ChatGPT. |
| ClaudeBot | Anthropic | Training. It collects content that may be used to train Anthropic’s models. |
| Claude-SearchBot | Anthropic | Search. It crawls to improve the search results Claude gives its users. |
| Claude-User | Anthropic | A user’s request. It fetches a page when someone asks Claude a question that needs it. |
| PerplexityBot | Perplexity | Search. It is used to show and link websites in Perplexity’s results. Perplexity says it is not used for training. |
| Perplexity-User | Perplexity | A user’s request. It visits a page to help answer a user’s question. |
| Google-Extended | Not a crawler. A name used only in robots.txt, to say whether Google may use your content to train Gemini models and to ground Gemini’s answers. |
Other companies run crawlers too. These four companies are the ones our free check looks at.
How a robots.txt rule works
robots.txt is a plain text file at the top of your site, at yoursite.com/robots.txt. It holds groups of rules. Each group starts with the name of a crawler and lists what that crawler may not fetch.
This file allows every crawler everywhere:
User-agent: *
Allow: /This one keeps the two training crawlers out and leaves search and user requests open:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /Two details of the standard matter here.
- A crawler follows the group with its own name. It falls back to the
*group only when no group names it. So a rule you wrote for all crawlers does not apply to a crawler that has its own group. - Names are matched without regard to capital letters.
gptbotandGPTBotare the same crawler.
Changes are not instant. OpenAI says its search systems can take about 24 hours to pick up a change to robots.txt.
What robots.txt cannot do
robots.txt is a request that well-behaved crawlers follow. The standard itself says its rules are not a form of access control, and Google’s documentation says the file cannot enforce anything: it is up to the crawler to obey.
Three limits follow from that.
- Fetches made for a user may not follow it. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the action. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its bots, Claude-User included, honour robots.txt.
- It does not keep a page out of Google. Google says a page that robots.txt disallows can still be indexed if other sites link to it.
- A firewall can overrule it. Your host or security service can refuse a crawler that your robots.txt welcomes. Nothing in the file will show it. Our guide to why AI assistants cannot read some websites describes how that happened to our own site.
Google is a special case
Google reads the web with Googlebot, the same crawler that builds Google Search. Google-Extended is not a second crawler. It is a name you can use in robots.txt, and it never appears as a visitor in your server’s logs.
Disallowing Google-Extended tells Google not to use your content to train future Gemini models or to ground answers in the Gemini apps. Google states that it does not affect whether your site appears in Google Search, and that it is not a ranking signal.
What about llms.txt?
llms.txt is a proposed file that lists a site’s main pages for AI tools. It is harmless, and it is not a way in.
Google’s guidance on AI features in Search says you do not need new machine-readable files to appear there, because Google Search does not use them. A study by Ahrefs looked at 137,210 domains. Of the roughly 38,000 that had a valid llms.txt file, 97% saw no requests for it at all in May 2026.
We publish one on our own site. We do not count it in our check, and we would not rely on it.
A sensible setup for a business site
If you want AI assistants to be able to find and quote your site, three things are worth doing.
- Allow the search crawlers. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the results those assistants link to.
- Allow the user-request fetchers. They are how an assistant opens your page when someone asks about you by name.
- Decide about training on its own. Refusing GPTBot, ClaudeBot or Google-Extended is a choice about how your content is used. It is separate from search: OpenAI says each of its settings is independent of the others, and Google says Google-Extended does not affect Google Search.
Then check that the answer your server gives matches the file. Open yoursite.com/robots.txt and read it. Our free AI check reads it for you, and also asks your server for the page under each crawler’s name.
Sources
Every figure and quoted fact in this guide was checked against these pages on 17 November 2026. Prices and product rules change, so check the source before you rely on a number.
- OpenAI: Overview of OpenAI CrawlersGPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot.
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?ClaudeBot, Claude-SearchBot and Claude-User.
- Perplexity: Perplexity crawlersPerplexityBot and Perplexity-User.
- Google Search Central: List of Google’s common crawlersGoogle-Extended.
- Google Search Central: Introduction to robots.txtWhat robots.txt can and cannot do.
- RFC 9309: Robots Exclusion ProtocolThe standard: where the file lives and how a crawler picks its group of rules.
- Google Search Central: Optimizing your website for generative AI features on Google SearchGoogle Search does not use llms.txt and similar files.
- Ahrefs: We Analyzed 137K Sites: 97% of llms.txt Files Never Get ReadPublished 15 June 2026.

