AI crawlers and robots.txt: what to allow or block

In short

AI companies run separate bots for training models, for building search indexes and for fetching pages a user asked about. You can block training and still appear in AI search answers by allowing the search bots, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, in robots.txt.

AI crawlers are the programs AI companies send to read websites. Most companies now run several, each with its own job. Some collect text to train future models. Some build the search index that AI answers draw from. Others fetch a page because a person just asked about it.

That split is what makes robots.txt useful. You can say no to training and still say yes to being found and cited. This guide lists the main crawlers as of November 2026 and gives you a snippet to copy.

Three columns of bot names. Training and data: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended. Search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot, Googlebot. User fetch: ChatGPT-User, Claude-User, Perplexity-User.
Three jobs. If you want to be cited, the middle column matters most.

What is robots.txt, in plain words?

It is a short text file at the root of your site, like https://www.example.com/robots.txt. It tells crawlers which parts of the site they may visit. Each block starts with one or more User-agent: lines naming a crawler, followed by Allow: or Disallow: rules.

Three things people often get wrong:

  • It only works on bots that choose to obey. Well-behaved crawlers follow it. Google itself says robots.txt "is not a mechanism for keeping a web page out of Google." Use noindex or a password for that.
  • Each host needs its own file. shop.example.com does not use the file on www.example.com, as Google and Bing both explain.
  • A named block replaces the catch-all. If a crawler finds a block with its own name, it follows that block and ignores the User-agent: * rules. Bing states this plainly for bingbot.

Which AI crawlers exist, and what does each one do?

Here are the main ones, grouped by company. Every row links to the company's own page.

Crawler (robots.txt name)CompanyJobFollows robots.txt?
GPTBotOpenAICollects content that may train modelsYes
OAI-SearchBotOpenAIFinds pages for ChatGPT searchYes
ChatGPT-UserOpenAIVisits a page because a user askedOpenAI says rules may not apply
ClaudeBotAnthropicCollects content that may train modelsYes
Claude-SearchBotAnthropicCrawls to improve Claude's search resultsYes
Claude-UserAnthropicFetches pages when people ask ClaudeYes
PerplexityBotPerplexityFinds and links pages in Perplexity searchYes
Perplexity-UserPerplexityFetches a page because a user askedPerplexity says it generally ignores robots.txt
Google-ExtendedGoogleA control token, not a separate crawler: covers Gemini training and grounding in Gemini AppsIt is itself a robots.txt rule
GooglebotGoogleGoogle Search, including AI Overviews and AI ModeYes
ApplebotAppleSearch features in Spotlight, Siri and SafariYes
Applebot-ExtendedAppleA control token: opts out of training Apple's foundation modelsIt is itself a robots.txt rule
bingbotMicrosoftBing search, which Copilot usesYes
CCBotCommon CrawlBuilds a free, open archive of web pages that anyone can downloadCommon Crawl documents how to block it

All names and roles are as of November 2026. Bots get added and renamed, so check the linked pages once or twice a year.

What is the difference between training bots and search bots?

Training bots collect pages that may be used to build future AI models. Blocking GPTBot or ClaudeBot does not remove you from ChatGPT or Claude search answers. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Anthropic says blocking ClaudeBot excludes your future content from its training data.

Search bots build the index that AI answers are drawn from. Blocking them takes you out of those answers. OpenAI is explicit that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

User-triggered fetchers visit one page because a person asked about it. OpenAI and Perplexity both say robots.txt may not apply to these visits.

Two tokens need extra care:

What should my robots.txt say?

This snippet allows AI search and user fetches, blocks the training and dataset bots listed above, and leaves normal search engines alone. Replace the sitemap address with your own.

# AI search and user-requested visits: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training and open datasets: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot, bingbot and Applebot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

We left Google-Extended out on purpose, because blocking it may also keep you out of Gemini app answers. If you want to refuse Gemini training anyway and accept that trade-off, add User-agent: Google-Extended to the training group.

If you are happy for your content to be used for training too, the simplest file is just this:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Before you change anything, keep a copy of your current file. If it already has Disallow lines for private areas like /admin/ or /checkout/, copy those lines into every group above. A crawler with its own group will not read the * group.

How do I check what my site allows today?

  1. Open https://yourdomain/robots.txt in a browser.
  2. Look for Disallow: / under any of the names in the table. That line blocks the whole site for that bot.
  3. Look for a User-agent: * group with Disallow: /. That blocks every bot without its own group.
  4. If your site is on a hosted platform, check its settings too. Some platforms write this file for you.

Our 15-minute checklist includes this step. Our free scanner reads your robots.txt and reports which of these bots are blocked from your homepage and main pages. It shows access rules; it cannot prove a bot actually visited. See our methodology for details.

Frequently asked questions

If I block GPTBot, will ChatGPT stop showing my site?

No. OpenAI treats GPTBot and OAI-SearchBot independently. To appear in ChatGPT search answers, OAI-SearchBot must be allowed.

How fast do changes take effect?

OpenAI says about 24 hours for its search systems. Google says recrawling can take from several days to several months. Other companies do not publish a figure that we could find.

Can I block an AI bot by IP address instead?

You can, but Anthropic warns that IP blocking may not work reliably for its bots, because they then cannot read your robots.txt. Robots.txt is the documented route.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says AI Overviews and AI Mode are controlled through Googlebot's rules. Google-Extended covers Gemini training and grounding in Gemini Apps, not Google Search.

Which bots does Copilot use?

Copilot searches Bing, so bingbot is the one to allow. See how each assistant finds sources.

Sources

  1. OpenAI: Overview of OpenAI crawlers
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Perplexity: Perplexity crawlers
  4. Google: Google's common crawlers (Google-Extended)
  5. Google Search Central: AI features and your website
  6. Apple Support: About Applebot
  7. Common Crawl: CCBot
  8. Bing Webmaster Blog: To crawl or not to crawl, that is BingBot's question
  9. Google Search Central: How to write and submit a robots.txt file
  10. Google Search Central: Introduction to robots.txt