Measure this visit?
Accept allows Google Analytics, Microsoft Clarity and PostHog, including session recording. Reject stops their recording. Cookieless counts and server request logs stay on. What we measure
API
Send a robots.txt file as text and a path. For each user agent, the answer says whether the rules allow that path, which group applied and which line decided. It is free and needs no key.
It runs the code of our robots.txt sandbox. Within the API's limits, the two give the same answer for the same file and path. It reads only the text you send. It never fetches a URL, so nobody can use it to probe a site.
Updated
https://synapsereality.io/api/robots/evaluateFree to usecurl -sS https://synapsereality.io/api/robots/evaluate \
-H 'Content-Type: application/json' \
-d '{"robots_txt":"User-agent: *\nDisallow: /private/\n\nUser-agent: ClaudeBot\nDisallow: /\n","path":"/private/report.html","user_agents":["ClaudeBot","Claude-SearchBot"]}'To send a file from disk, let jq build the JSON. It escapes the line breaks for you.
jq -n --rawfile robots robots.txt \
'{robots_txt: $robots, path: "/blog/"}' |
curl -sS https://synapsereality.io/api/robots/evaluate \
-H 'Content-Type: application/json' --data-binary @-From a web page or a script, fetch works the same way. Any site can call it.
const res = await fetch('https://synapsereality.io/api/robots/evaluate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ robots_txt: robotsTxt, path: '/blog/' }),
})
const answer = await res.json()A JSON object with these fields. Any other field is refused.
robots_txt string, requiredpath string, required/blog/post, up to 2,048 characters. A full URL works too, and then its path and query are tested.user_agents list of strings, optionalThis is the answer to the curl example above, as the API's own code returns it.
{
"ok": true,
"path": "/private/report.html",
"groups": 2,
"truncated": false,
"results": [
{
"user_agent": "ClaudeBot",
"token": "claudebot",
"allowed": false,
"group": "claudebot",
"rule": {
"line": 5,
"text": "Disallow: /",
"allow": false,
"pattern": "/",
"truncated": false
}
},
{
"user_agent": "Claude-SearchBot",
"token": "claude-searchbot",
"allowed": false,
"group": "*",
"rule": {
"line": 2,
"text": "Disallow: /private/",
"allow": false,
"pattern": "/private/",
"truncated": false
}
}
],
"matcher": {
"package": "answer-crawler-check",
"version": "0.1.2"
}
}pathgroupstruncatedresultsallowed is the verdict. group names the group that applied. It is * when the agent fell through to the group for all crawlers, and null when the file has neither. rule is the line that decided, with its line number, or null when no rule matched.matcherEvery error the API sends is JSON with ok set to false, a code to branch on and an error sentence to show a person. The refusals our web server sends itself, listed under limits, are HTML.
bad_json, bad_inputmethod_not_allowedtoo_largerobots_txt is over its cap.too_much_workrate_limitedRetry-After says how many seconds to wait.internalbusy, unavailableX-RateLimit-Limit, and what is left this hour in X-RateLimit-Remaining.robots_txt up to 500 KiB. Send the text as UTF-8. Escaping each letter outside ASCII as \u00e9 makes the body up to six times larger.truncated field says so./api/ requests a minute, after a burst of ten. Past that, and for any body over 1 MiB, it answers itself with an HTML page and no CORS header. A page calling the API from another site then sees a failed request with no status.52 of 55 vectors pass through this endpoint's code.
Each of our RFC 9309 test vectors is a robots.txt file, a crawler name, a path and the result the standard gives. Every build of this page runs all 55 through the API's code and prints the count it measures. The CI run of our server tests sends them through the endpoint itself.
The 3 that fail are the cases where release 0.1.2 of the matcher reads a file another way than RFC 9309 does. Their ids are other-record-keeps-names-in-one-group-first, unreserved-escape-in-the-path and unreserved-escape-figure-4. The vectors page explains each one, and the sandbox shows the same results.
The full description is an OpenAPI 3.1 file at /tools/robots-sandbox/api/openapi.json. Here is its request schema.
{
"type": "object",
"additionalProperties": false,
"required": [
"robots_txt",
"path"
],
"properties": {
"robots_txt": {
"type": "string",
"description": "The file as text. At most 512000 bytes of UTF-8."
},
"path": {
"type": "string",
"maxLength": 2048,
"pattern": "^[^\\u0000-\\u001f\\u007f]*$",
"description": "A path such as /blog/post, or a full URL whose path and query are tested. No control characters."
},
"user_agents": {
"type": "array",
"minItems": 1,
"maxItems": 50,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 200,
"pattern": "^[^\\u0000-\\u001f\\u007f]*[^\\s\\u0000-\\u001f\\u007f][^\\u0000-\\u001f\\u007f]*$",
"description": "Not blank, and no control characters."
},
"description": "Names to test. Left out, the API tests the 9 AI crawler names the sandbox tests.",
"default": [
"ClaudeBot",
"Claude-SearchBot",
"Claude-User",
"GPTBot",
"OAI-SearchBot",
"ChatGPT-User",
"Google-Extended",
"PerplexityBot",
"Perplexity-User"
]
}
}
}X-RateLimit-Limit header always gives the hourly limit in force.