Tool

robots.txt sandbox: test a path against each AI crawler

Paste a robots.txt file and type a path. For each AI crawler name in our table you see whether the rules allow that path, which group applied and which line decided. It runs in your browser. Nothing you paste is sent, fetched or stored on a server.

The sandbox reads the rules as the standard, RFC 9309, defines them. It does not show what any engine actually does.

Updated

The file and the path

What the rules say

Paste a file or load an example. The result shows here as you type.

How the rules are read

These are the rules of RFC 9309, the Robots Exclusion Protocol, as our command-line tool applies them.

  • A crawler follows the group that names it. A name matches in any letter case.
  • A crawler that no group names follows the group for *. A crawler with a group of its own takes nothing from the * group.
  • Groups that name the same crawler count as one group.
  • Inside the group, the longest pattern that matches the path decides. When an allow rule and a disallow rule are equally long, the allow rule decides.
  • In a pattern, * stands for any run of characters. A $ at the end means the path must end there.
  • A path that no rule matches is allowed. So is every path when no group applies.
  • A path is compared with its letter case kept, so /Blog and /blog are two paths.
  • /robots.txt itself is always allowed.

The table has 9 names from our AI crawler tokens table. They are the names for model training, for search and for fetches a user asks for. We checked that table against each vendor's page on 8 Nov 2026.

  • Model training: ClaudeBot, GPTBot, Google-Extended
  • Search index: Claude-SearchBot, OAI-SearchBot, PerplexityBot
  • Fetches a user asks for: Claude-User, ChatGPT-User, Perplexity-User

The sandbox leaves out OAI-AdsBot, which that table files under another role.

Those are the names Anthropic, OpenAI, Google and Perplexity document for AI. No other crawler has a row, and that includes a search engine's main crawler.

An old name in your file is flagged with the names that table lists in its place.

What it cannot tell you

It cannot tell you what a crawler does. It applies the standard to the text you paste, and a crawler may read the same file in its own way. Each vendor's page, linked from the tokens table, says what its names control.

It does not fetch your site, so it tests the text in the box and not the file your server returns. It cannot see a firewall rule or a bot challenge, and either can turn away a crawler that robots.txt allows. It cannot see a noindex header, which keeps an allowed page out of an index. The instant audit fetches one page from our server and reads what a crawler receives. The response header reader explains response headers you paste, such as an X-Robots-Tag line or a redirect chain.

Where it differs from the standard

A review found two cases where release 0.1.2 of the matcher reads a file another way than section 2.2 of RFC 9309 does. The sandbox shows what the matcher returns, so it has both.

  • An escape for a plain character is compared as written. A rule for /private does not match the path /%70rivate, and the standard says it should.
  • A line that is not a rule, such as Crawl-delay, ends a run of crawler names. The names above that line then get none of the rules below it. The standard keeps them in one group.

Our RFC 9309 test vectors hold both cases, with the result the standard gives, so any parser can be tested on them. That page also lists two more cases this release does not pass.

What happens to the text you paste

It stays in your browser. The sandbox makes no request with it, and closing or reloading the tab discards it. The session recording and the click analytics on the rest of this site read text off a page, so this page does not load them.

A link you copy holds the file and the path after the # sign, and browsers do not send that part of an address to a server. This page takes that part off the address before any other script on it runs. A link carries a file of up to 8 KiB. A longer file is tested as usual, with no link.

Test every URL in a sitemap

The sandbox tests one path at a time. answer-crawler-check runs the same rules over every URL in your sitemap, from a terminal or a CI job. The sandbox uses the reader and the matcher of its release 0.1.2.

npx answer-crawler-check https://example.com/sitemap.xml