Measure this visit?
Accept allows Google Analytics, Microsoft Clarity and PostHog, including session recording. Reject stops their recording. Cookieless counts and server request logs stay on. What we measure
Tool
Paste a robots.txt file and type a path. For each AI crawler name in our table you see whether the rules allow that path, which group applied and which line decided. It runs in your browser. Nothing you paste is sent, fetched or stored on a server.
The sandbox reads the rules as the standard, RFC 9309, defines them. It does not show what any engine actually does.
Updated
Paste a file or load an example. The result shows here as you type.
These are the rules of RFC 9309, the Robots Exclusion Protocol, as our command-line tool applies them.
*. A crawler with a group of its own takes nothing from the * group.* stands for any run of characters. A $ at the end means the path must end there./Blog and /blog are two paths./robots.txt itself is always allowed.The table has 9 names from our AI crawler tokens table. They are the names for model training, for search and for fetches a user asks for. We checked that table against each vendor's page on 8 Nov 2026.
ClaudeBot, GPTBot, Google-ExtendedClaude-SearchBot, OAI-SearchBot, PerplexityBotClaude-User, ChatGPT-User, Perplexity-UserThe sandbox leaves out OAI-AdsBot, which that table files under another role.
Those are the names Anthropic, OpenAI, Google and Perplexity document for AI. No other crawler has a row, and that includes a search engine's main crawler.
An old name in your file is flagged with the names that table lists in its place.
It cannot tell you what a crawler does. It applies the standard to the text you paste, and a crawler may read the same file in its own way. Each vendor's page, linked from the tokens table, says what its names control.
It does not fetch your site, so it tests the text in the box and not the file your server returns. It cannot see a firewall rule or a bot challenge, and either can turn away a crawler that robots.txt allows. It cannot see a noindex header, which keeps an allowed page out of an index. The instant audit fetches one page from our server and reads what a crawler receives. The response header reader explains response headers you paste, such as an X-Robots-Tag line or a redirect chain.
A review found two cases where release 0.1.2 of the matcher reads a file another way than section 2.2 of RFC 9309 does. The sandbox shows what the matcher returns, so it has both.
/private does not match the path /%70rivate, and the standard says it should.Crawl-delay, ends a run of crawler names. The names above that line then get none of the rules below it. The standard keeps them in one group.Our RFC 9309 test vectors hold both cases, with the result the standard gives, so any parser can be tested on them. That page also lists two more cases this release does not pass.
It stays in your browser. The sandbox makes no request with it, and closing or reloading the tab discards it. The session recording and the click analytics on the rest of this site read text off a page, so this page does not load them.
A link you copy holds the file and the path after the # sign, and browsers do not send that part of an address to a server. This page takes that part off the address before any other script on it runs. A link carries a file of up to 8 KiB. A longer file is tested as usual, with no link.
The sandbox tests one path at a time. answer-crawler-check runs the same rules over every URL in your sitemap, from a terminal or a CI job. The sandbox uses the reader and the matcher of its release 0.1.2.
npx answer-crawler-check https://example.com/sitemap.xml