// open source reference

Which AI crawler names belong in your robots.txt

Answer four yes or no questions and paste the robots.txt lines under each answer. Every name comes from the crawler tokens table, which lists what OpenAI, Anthropic, Google and Perplexity document. Their pages were read on .

Updated

Question 1 of 4

Do you want every AI crawler kept out?

This means all three kinds of crawler in the table: model training, AI search and fetches a person asks for. A robots.txt rule asks a crawler to stay out. It does not lock a page. Read first what the vendors say about a block on fetches a person asks for.

Claude-User Anthropic
Fetches pages when a person asks Claude a question.
If you block it Anthropic says Claude can no longer retrieve your pages in answer to a user's query.
ChatGPT-User OpenAI
Visits pages for actions a user takes in ChatGPT and Custom GPTs.
If you block it OpenAI says robots.txt rules may not apply to it, because a user started the request. It is not used to decide whether content appears in search.
Perplexity-User Perplexity
Visits a page when a user asks Perplexity a question, and can link it in the answer.
If you block it Perplexity says this fetcher generally ignores robots.txt rules, because a user requested the fetch.

Yes, ask them all to stay out

Paste this and stop here. It has a group for every name in questions 2, 3 and 4. The notes above say how far a block reaches for a fetch a person asks for.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

No, some are welcome

Answer the next three questions. Each one stands alone, so a yes to one does not change the others.

Question 2 of 4

Do you want your pages kept out of model training?

These are the model training names in the table, with what each vendor says about its own.

ClaudeBot Anthropic
Collects web content that could contribute to training Anthropic's models.
If you block it Anthropic says it signals that the site's future materials should be excluded from its training datasets.
GPTBot OpenAI
Crawls content that may be used to train OpenAI's foundation models.
If you block it OpenAI says it indicates the content should not be used for training. Each OpenAI setting is independent of the others.
Google-Extended Google
Whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
If you block it Google says it has no effect on inclusion in Google Search and is not a ranking signal. It has no user agent string of its own. Google crawls with its existing user agents.

The table has no training name for Perplexity.

Yes, no training

Paste this. It tells each training crawler to stay out of the whole site.

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

No, training is fine

Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.

User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

Question 3 of 4

These are the search index names in the table, with what each vendor says about its own.

Claude-SearchBot Anthropic
Crawls to improve the quality of Claude's search results.
If you block it Anthropic says it stops your content being indexed for search, which can reduce your visibility in Claude's search results.
OAI-SearchBot OpenAI
Surfaces websites in ChatGPT's search features.
If you block it OpenAI says opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links.
PerplexityBot Perplexity
Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
If you block it Perplexity recommends allowing it for your site to appear in its search results.

The table has no search name for Google.

Yes, stay in AI search

Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.

User-agent: Claude-SearchBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

No, stay out of AI search

Paste this. The notes above carry what each vendor says about blocking its search crawler.

User-agent: Claude-SearchBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Question 4 of 4

Do you want to allow fetches a person asks for?

These are the names in the table for a fetch a person asks for, with what each vendor says about its own.

Claude-User Anthropic
Fetches pages when a person asks Claude a question.
If you block it Anthropic says Claude can no longer retrieve your pages in answer to a user's query.
ChatGPT-User OpenAI
Visits pages for actions a user takes in ChatGPT and Custom GPTs.
If you block it OpenAI says robots.txt rules may not apply to it, because a user started the request. It is not used to decide whether content appears in search.
Perplexity-User Perplexity
Visits a page when a user asks Perplexity a question, and can link it in the answer.
If you block it Perplexity says this fetcher generally ignores robots.txt rules, because a user requested the fetch.

The table has no user name for Google.

Yes, allow those fetches

Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.

User-agent: Claude-User
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

No, block those fetches

Paste this, and read the notes above first. They carry what each vendor says about blocking this kind of fetch.

User-agent: Claude-User
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Perplexity-User
Disallow: /

Put your answers in one file

Paste the pieces you chose into robots.txt, in any order. Each name appears under one question only, so the pieces never conflict with each other. The panel at the end of this page joins them for you.

If your file already has a group for one of these names, edit that group. Do not add a second one. RFC 9309 reads two groups for one name as one group, and an old line can outweigh the new one (section 2.2.2).

robots.txt is matched by name. RFC 9309, the robots.txt standard, says a crawler obeys the group that matches its own name. With no such group it obeys the * group, if there is one (section 2.2.1).

Then test the file. answer-crawler-check reads every URL in your sitemap and reports which answer crawlers your robots.txt blocks. To try one path first, paste the file into the robots.txt sandbox.

npx answer-crawler-check https://example.com/sitemap.xml

Training crawlers are left out of that check by default. Name them with --agents to check them too.

Names this page leaves out

  • OAI-AdsBot from OpenAI is in no snippet. Visits only landing pages submitted as ads on ChatGPT, to check them against OpenAI's policies.
  • anthropic-ai is an old name. Not on Anthropic's crawler page, which names three bots.
  • Claude-Web is an old name. Not on Anthropic's crawler page, which names three bots.

The crawler tokens page shows which names to use in place of an old one. This page covers the four vendors in that table and has no data on any other crawler.

Where the names come from

Every name and every note above is a row of the crawler tokens table. The rows come from these vendor pages, read on 8 Nov 2026. The same rows are at crawler-tokens.json.

Your robots.txt

Pick an answer to each question here, and this panel joins their snippets into one file. It keeps the order of the questions and gives each name one group.

Test every URL in your sitemap

Once the file is on your site, run this with your own sitemap address.

npx answer-crawler-check https://example.com/sitemap.xml