Measure this visit?
Accept allows Google Analytics, Microsoft Clarity and PostHog, including session recording. Reject stops their recording. Cookieless counts and server request logs stay on. What we measure
// open source reference
Answer four yes or no questions and paste the robots.txt lines under each answer. Every name comes from the crawler tokens table, which lists what OpenAI, Anthropic, Google and Perplexity document. Their pages were read on .
Updated
Question 1 of 4
This means all three kinds of crawler in the table: model training, AI search and fetches a person asks for. A robots.txt rule asks a crawler to stay out. It does not lock a page. Read first what the vendors say about a block on fetches a person asks for.
Claude-User AnthropicChatGPT-User OpenAIPerplexity-User PerplexityPaste this and stop here. It has a group for every name in questions 2, 3 and 4. The notes above say how far a block reaches for a fetch a person asks for.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /Answer the next three questions. Each one stands alone, so a yes to one does not change the others.
Question 2 of 4
These are the model training names in the table, with what each vendor says about its own.
ClaudeBot AnthropicGPTBot OpenAIGoogle-Extended GoogleThe table has no training name for Perplexity.
Paste this. It tells each training crawler to stay out of the whole site.
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.
User-agent: ClaudeBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /Question 3 of 4
These are the search index names in the table, with what each vendor says about its own.
Claude-SearchBot AnthropicOAI-SearchBot OpenAIPerplexityBot PerplexityThe table has no search name for Google.
Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.
User-agent: Claude-SearchBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Paste this. The notes above carry what each vendor says about blocking its search crawler.
User-agent: Claude-SearchBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /Question 4 of 4
These are the names in the table for a fetch a person asks for, with what each vendor says about its own.
Claude-User AnthropicChatGPT-User OpenAIPerplexity-User PerplexityThe table has no user name for Google.
Paste this. Each name gets a group of its own, so a stricter rule for all crawlers no longer applies to it.
User-agent: Claude-User
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Perplexity-User
Allow: /Paste this, and read the notes above first. They carry what each vendor says about blocking this kind of fetch.
User-agent: Claude-User
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Perplexity-User
Disallow: /Paste the pieces you chose into robots.txt, in any order. Each name appears under one question only, so the pieces never conflict with each other. The panel at the end of this page joins them for you.
If your file already has a group for one of these names, edit that group. Do not add a second one. RFC 9309 reads two groups for one name as one group, and an old line can outweigh the new one (section 2.2.2).
robots.txt is matched by name. RFC 9309, the robots.txt standard, says a crawler obeys the group that matches its own name. With no such group it obeys the * group, if there is one (section 2.2.1).
Then test the file. answer-crawler-check reads every URL in your sitemap and reports which answer crawlers your robots.txt blocks. To try one path first, paste the file into the robots.txt sandbox.
npx answer-crawler-check https://example.com/sitemap.xmlTraining crawlers are left out of that check by default. Name them with --agents to check them too.
OAI-AdsBot from OpenAI is in no snippet. Visits only landing pages submitted as ads on ChatGPT, to check them against OpenAI's policies.anthropic-ai is an old name. Not on Anthropic's crawler page, which names three bots.Claude-Web is an old name. Not on Anthropic's crawler page, which names three bots.The crawler tokens page shows which names to use in place of an old one. This page covers the four vendors in that table and has no data on any other crawler.
Every name and every note above is a row of the crawler tokens table. The rows come from these vendor pages, read on 8 Nov 2026. The same rows are at crawler-tokens.json.
Pick an answer to each question here, and this panel joins their snippets into one file. It keeps the order of the questions and gives each name one group.
Your file shows here once you pick an answer that has a snippet.
Once the file is on your site, run this with your own sitemap address.
npx answer-crawler-check https://example.com/sitemap.xml