API

robots.txt evaluation API

Send a robots.txt file as text and a path. For each user agent, the answer says whether the rules allow that path, which group applied and which line decided. It is free and needs no key.

It runs the code of our robots.txt sandbox. Within the API's limits, the two give the same answer for the same file and path. It reads only the text you send. It never fetches a URL, so nobody can use it to probe a site.

Updated

POSThttps://synapsereality.io/api/robots/evaluateFree to use

Try it with curl

curl -sS https://synapsereality.io/api/robots/evaluate \
  -H 'Content-Type: application/json' \
  -d '{"robots_txt":"User-agent: *\nDisallow: /private/\n\nUser-agent: ClaudeBot\nDisallow: /\n","path":"/private/report.html","user_agents":["ClaudeBot","Claude-SearchBot"]}'

To send a file from disk, let jq build the JSON. It escapes the line breaks for you.

jq -n --rawfile robots robots.txt \
  '{robots_txt: $robots, path: "/blog/"}' |
  curl -sS https://synapsereality.io/api/robots/evaluate \
    -H 'Content-Type: application/json' --data-binary @-

From a web page or a script, fetch works the same way. Any site can call it.

const res = await fetch('https://synapsereality.io/api/robots/evaluate', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ robots_txt: robotsTxt, path: '/blog/' }),
})
const answer = await res.json()

Request

A JSON object with these fields. Any other field is refused.

robots_txt string, required
The file as text, at most 512,000 bytes once encoded as UTF-8 (500 KiB). A blank file has no rules, so the API allows every agent, as RFC 9309 does. The sandbox waits for a file instead.
path string, required
The path to test, such as /blog/post, up to 2,048 characters. A full URL works too, and then its path and query are tested.
user_agents list of strings, optional
Up to 50 names of up to 200 characters each. A full user agent string works, since only its product token is matched. Left out, the API tests the 9 AI crawler names the sandbox tests, from our AI crawler tokens table.

Response

This is the answer to the curl example above, as the API's own code returns it.

{
  "ok": true,
  "path": "/private/report.html",
  "groups": 2,
  "truncated": false,
  "results": [
    {
      "user_agent": "ClaudeBot",
      "token": "claudebot",
      "allowed": false,
      "group": "claudebot",
      "rule": {
        "line": 5,
        "text": "Disallow: /",
        "allow": false,
        "pattern": "/",
        "truncated": false
      }
    },
    {
      "user_agent": "Claude-SearchBot",
      "token": "claude-searchbot",
      "allowed": false,
      "group": "*",
      "rule": {
        "line": 2,
        "text": "Disallow: /private/",
        "allow": false,
        "pattern": "/private/",
        "truncated": false
      }
    }
  ],
  "matcher": {
    "package": "answer-crawler-check",
    "version": "0.1.2"
  }
}
path
The path that was tested, written as a browser would request it. A space or a letter outside ASCII is percent-encoded.
groups
How many User-agent names the file holds. Zero means no rule applies to anyone.
truncated
True when the file holds more than 20,000 rules. Only the first 20,000 were stored, so treat the answer with care. RFC 9309 merges every group that names an agent. A name seen before that point keeps the rules stored under it. A name first seen after it still counts as a group, with no rules.
results
One entry for each user agent, in the order you sent them. allowed is the verdict. group names the group that applied. It is * when the agent fell through to the group for all crawlers, and null when the file has neither. rule is the line that decided, with its line number, or null when no rule matched.
matcher
The release of answer-crawler-check whose reader and matcher gave the answer.

Errors

Every error the API sends is JSON with ok set to false, a code to branch on and an error sentence to show a person. The refusals our web server sends itself, listed under limits, are HTML.

400 bad_json, bad_input
The body is not UTF-8 JSON, a field is missing or has the wrong type, or an unknown field was sent. Also a path or a name past its limit or holding a control character, and a blank name.
405 method_not_allowed
Any method but POST and OPTIONS.
413 too_large
The body is over 1,000,000 bytes, or robots_txt is over its cap.
413 too_much_work
The file and the path are past the sandbox's work budget, or the test ran past 400 ms.
429 rate_limited
More than the hourly limit from one address. The error names the limit, and Retry-After says how many seconds to wait.
500 internal
Something failed at our end. The answer holds nothing you sent.
503 busy, unavailable
The evaluator or our request counter is busy or down. Try again after a few seconds.

Limits

  • 60 requests an hour from one IP address. For IPv6 that means one /64 network. Each request the limit counts gets the limit in force back in X-RateLimit-Limit, and what is left this hour in X-RateLimit-Remaining.
  • A request body of up to 1,000,000 bytes, with robots_txt up to 500 KiB. Send the text as UTF-8. Escaping each letter outside ASCII as \u00e9 makes the body up to six times larger.
  • A path of up to 2,048 characters, and up to 50 user agents.
  • A deciding pattern longer than 1,024 characters is cut, and its truncated field says so.
  • Our web server also limits each address to ten /api/ requests a minute, after a burst of ten. Past that, and for any body over 1 MiB, it answers itself with an HTML page and no CORS header. A page calling the API from another site then sees a failed request with no status.

Tested against the RFC 9309 test vectors

52 of 55 vectors pass through this endpoint's code.

Each of our RFC 9309 test vectors is a robots.txt file, a crawler name, a path and the result the standard gives. Every build of this page runs all 55 through the API's code and prints the count it measures. The CI run of our server tests sends them through the endpoint itself.

The 3 that fail are the cases where release 0.1.2 of the matcher reads a file another way than RFC 9309 does. Their ids are other-record-keeps-names-in-one-group-first, unreserved-escape-in-the-path and unreserved-escape-figure-4. The vectors page explains each one, and the sandbox shows the same results.

OpenAPI

The full description is an OpenAPI 3.1 file at /tools/robots-sandbox/api/openapi.json. Here is its request schema.

{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "robots_txt",
    "path"
  ],
  "properties": {
    "robots_txt": {
      "type": "string",
      "description": "The file as text. At most 512000 bytes of UTF-8."
    },
    "path": {
      "type": "string",
      "maxLength": 2048,
      "pattern": "^[^\\u0000-\\u001f\\u007f]*$",
      "description": "A path such as /blog/post, or a full URL whose path and query are tested. No control characters."
    },
    "user_agents": {
      "type": "array",
      "minItems": 1,
      "maxItems": 50,
      "items": {
        "type": "string",
        "minLength": 1,
        "maxLength": 200,
        "pattern": "^[^\\u0000-\\u001f\\u007f]*[^\\s\\u0000-\\u001f\\u007f][^\\u0000-\\u001f\\u007f]*$",
        "description": "Not blank, and no control characters."
      },
      "description": "Names to test. Left out, the API tests the 9 AI crawler names the sandbox tests.",
      "default": [
        "ClaudeBot",
        "Claude-SearchBot",
        "Claude-User",
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "Google-Extended",
        "PerplexityBot",
        "Perplexity-User"
      ]
    }
  }
}

Terms

  • The API is free to use from any site, script or CI job. It needs no key and no sign-up.
  • We do not log or store the robots.txt text, the path or the names you send. Each request is read once, answered and dropped. A body over 16 KiB passes through a temporary file on our web server, which deletes it when the request ends.
  • To count requests, we keep a counter for each address under a code made from the address and a secret key. The table holds no address. A counter is deleted about a day after its last use, and copies in our database backups are kept for up to two years.
  • The answer says what the rules mean under RFC 9309, as our matcher reads them. It does not say what any crawler will do, and a crawler can read the same file in its own way.
  • We give no promise of uptime. If a limit changes, this page changes with it, since it reads every limit from the file the API enforces. The X-RateLimit-Limit header always gives the hourly limit in force.