// open source reference
AI crawler tokens: the old names, and what each vendor documents today
A robots.txt that names only anthropic-ai or Claude-Web names bots that Anthropic's crawler page doesn't list. Here is every token OpenAI, Anthropic, Google and Perplexity document for their AI crawlers, what each one controls, and the page it comes from. All four pages were read on .
Updated
Names to replace
| In your file | Status | Tokens the vendor lists | Source |
|---|---|---|---|
anthropic-ai | Not on Anthropic's crawler page, which names three bots | ClaudeBot, Claude-SearchBot, Claude-User | Anthropic, checked 8 Nov 2026 |
Claude-Web | Not on Anthropic's crawler page, which names three bots | ClaudeBot, Claude-SearchBot, Claude-User | Anthropic, checked 8 Nov 2026 |
Why an old name does nothing for the new bots: robots.txt is matched by name. RFC 9309, the robots.txt standard, says a crawler uses its own product token "to find the group that matches" and obeys that group. If no group matches, it obeys the * group, if there is one (section 2.2.1). A group for anthropic-ai doesn't match ClaudeBot.
If what you meant was "no training, but stay in AI search", these are the training tokens the four vendors document. Perplexity documents none, and says PerplexityBot isn't used to crawl content for foundation models.
Google-Extended covers more than training. Google's page says it also controls grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Blocking it does not change Google Search.
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Every token, by vendor
The "What it controls" and "If you block it" columns say what the vendor's page says, in shorter words. Nothing here is about what an engine does beyond its own documentation.
Anthropic
Source: Does Anthropic crawl data from the web, and how can site owners block the crawler?, checked 8 Nov 2026.
| Token | Used for | What it controls | If you block it |
|---|---|---|---|
ClaudeBot | Model training | Collects web content that could contribute to training Anthropic's models. | Anthropic says it signals that the site's future materials should be excluded from its training datasets. |
Claude-SearchBot | Search index | Crawls to improve the quality of Claude's search results. | Anthropic says it stops your content being indexed for search, which can reduce your visibility in Claude's search results. |
Claude-User | Fetch for a user | Fetches pages when a person asks Claude a question. | Anthropic says Claude can no longer retrieve your pages in answer to a user's query. |
OpenAI
Source: Overview of OpenAI Crawlers, checked 8 Nov 2026.
| Token | Used for | What it controls | If you block it |
|---|---|---|---|
GPTBot | Model training | Crawls content that may be used to train OpenAI's foundation models. | OpenAI says it indicates the content should not be used for training. Each OpenAI setting is independent of the others. |
OAI-SearchBot | Search index | Surfaces websites in ChatGPT's search features. | OpenAI says opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links. |
ChatGPT-User | Fetch for a user | Visits pages for actions a user takes in ChatGPT and Custom GPTs. | OpenAI says robots.txt rules may not apply to it, because a user started the request. It is not used to decide whether content appears in search. |
OAI-AdsBot | Ads review | Visits only landing pages submitted as ads on ChatGPT, to check them against OpenAI's policies. | OpenAI says its data is not used to train foundation models. |
Source: Google's common crawlers, checked 8 Nov 2026.
| Token | Used for | What it controls | If you block it |
|---|---|---|---|
Google-Extended | Model training | Whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. | Google says it has no effect on inclusion in Google Search and is not a ranking signal. It has no user agent string of its own. Google crawls with its existing user agents. |
Perplexity
Source: Perplexity Crawlers, checked 8 Nov 2026.
| Token | Used for | What it controls | If you block it |
|---|---|---|---|
PerplexityBot | Search index | Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models. | Perplexity recommends allowing it for your site to appear in its search results. |
Perplexity-User | Fetch for a user | Visits a page when a user asks Perplexity a question, and can link it in the answer. | Perplexity says this fetcher generally ignores robots.txt rules, because a user requested the fetch. |
The table as JSON
The same rows, with the source links and the date, are at crawler-tokens.json. When a vendor changes its page, the date moves with the rows.
Test your own robots.txt
answer-crawler-check reads every URL in your sitemap and reports which answer crawlers your robots.txt blocks, with the rule and line that blocked each one.
npx answer-crawler-check https://example.com/sitemap.xmlBy default it checks the crawlers that fetch pages for answers or build the index answers cite. Training crawlers such as GPTBot are left out. Name them with --agents to check them too.