AI crawlers and robots.txt: what to allow or block
In short
AI companies run separate bots for training models, for building search indexes and for fetching pages a user asked about. You can block training and still appear in AI search answers by allowing the search bots, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, in robots.txt.
AI crawlers are the programs AI companies send to read websites. Most companies now run several, each with its own job. Some collect text to train future models. Some build the search index that AI answers draw from. Others fetch a page because a person just asked about it.
That split is what makes robots.txt useful. You can say no to training and still say yes to being found and cited. This guide lists the main crawlers as of November 2026 and gives you a snippet to copy.
What is robots.txt, in plain words?
It is a short text file at the root of your site, like https://www.example.com/robots.txt. It tells crawlers which parts of the site they may visit. Each block starts with one or more User-agent: lines naming a crawler, followed by Allow: or Disallow: rules.
Three things people often get wrong:
- It only works on bots that choose to obey. Well-behaved crawlers follow it. Google itself says robots.txt "is not a mechanism for keeping a web page out of Google." Use
noindexor a password for that. - Each host needs its own file.
shop.example.comdoes not use the file onwww.example.com, as Google and Bing both explain. - A named block replaces the catch-all. If a crawler finds a block with its own name, it follows that block and ignores the
User-agent: *rules. Bing states this plainly for bingbot.
Which AI crawlers exist, and what does each one do?
Here are the main ones, grouped by company. Every row links to the company's own page.
| Crawler (robots.txt name) | Company | Job | Follows robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may train models | Yes |
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | Yes |
| ChatGPT-User | OpenAI | Visits a page because a user asked | OpenAI says rules may not apply |
| ClaudeBot | Anthropic | Collects content that may train models | Yes |
| Claude-SearchBot | Anthropic | Crawls to improve Claude's search results | Yes |
| Claude-User | Anthropic | Fetches pages when people ask Claude | Yes |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity search | Yes |
| Perplexity-User | Perplexity | Fetches a page because a user asked | Perplexity says it generally ignores robots.txt |
| Google-Extended | A control token, not a separate crawler: covers Gemini training and grounding in Gemini Apps | It is itself a robots.txt rule | |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Applebot | Apple | Search features in Spotlight, Siri and Safari | Yes |
| Applebot-Extended | Apple | A control token: opts out of training Apple's foundation models | It is itself a robots.txt rule |
| bingbot | Microsoft | Bing search, which Copilot uses | Yes |
| CCBot | Common Crawl | Builds a free, open archive of web pages that anyone can download | Common Crawl documents how to block it |
All names and roles are as of November 2026. Bots get added and renamed, so check the linked pages once or twice a year.
What is the difference between training bots and search bots?
Training bots collect pages that may be used to build future AI models. Blocking GPTBot or ClaudeBot does not remove you from ChatGPT or Claude search answers. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Anthropic says blocking ClaudeBot excludes your future content from its training data.
Search bots build the index that AI answers are drawn from. Blocking them takes you out of those answers. OpenAI is explicit that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
User-triggered fetchers visit one page because a person asked about it. OpenAI and Perplexity both say robots.txt may not apply to these visits.
Two tokens need extra care:
- Google-Extended covers more than training. Google lists "Grounding in Gemini Apps" among its uses. Blocking it does not affect Google Search or AI Overviews, but it may affect answers in the Gemini app.
- Applebot-Extended does not crawl anything. Apple says pages that disallow it "can still be included in search results."
What should my robots.txt say?
This snippet allows AI search and user fetches, blocks the training and dataset bots listed above, and leaves normal search engines alone. Replace the sitemap address with your own.
# AI search and user-requested visits: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Model training and open datasets: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including Googlebot, bingbot and Applebot
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
We left Google-Extended out on purpose, because blocking it may also keep you out of Gemini app answers. If you want to refuse Gemini training anyway and accept that trade-off, add User-agent: Google-Extended to the training group.
If you are happy for your content to be used for training too, the simplest file is just this:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Before you change anything, keep a copy of your current file. If it already has Disallow lines for private areas like /admin/ or /checkout/, copy those lines into every group above. A crawler with its own group will not read the * group.
How do I check what my site allows today?
- Open
https://yourdomain/robots.txtin a browser. - Look for
Disallow: /under any of the names in the table. That line blocks the whole site for that bot. - Look for a
User-agent: *group withDisallow: /. That blocks every bot without its own group. - If your site is on a hosted platform, check its settings too. Some platforms write this file for you.
Our 15-minute checklist includes this step. Our free scanner reads your robots.txt and reports which of these bots are blocked from your homepage and main pages. It shows access rules; it cannot prove a bot actually visited. See our methodology for details.
Frequently asked questions
If I block GPTBot, will ChatGPT stop showing my site?
No. OpenAI treats GPTBot and OAI-SearchBot independently. To appear in ChatGPT search answers, OAI-SearchBot must be allowed.
How fast do changes take effect?
OpenAI says about 24 hours for its search systems. Google says recrawling can take from several days to several months. Other companies do not publish a figure that we could find.
Can I block an AI bot by IP address instead?
You can, but Anthropic warns that IP blocking may not work reliably for its bots, because they then cannot read your robots.txt. Robots.txt is the documented route.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says AI Overviews and AI Mode are controlled through Googlebot's rules. Google-Extended covers Gemini training and grounding in Gemini Apps, not Google Search.
Which bots does Copilot use?
Copilot searches Bing, so bingbot is the one to allow. See how each assistant finds sources.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: Google's common crawlers (Google-Extended)
- Google Search Central: AI features and your website
- Apple Support: About Applebot
- Common Crawl: CCBot
- Bing Webmaster Blog: To crawl or not to crawl, that is BingBot's question
- Google Search Central: How to write and submit a robots.txt file
- Google Search Central: Introduction to robots.txt