Measure this visit?
Accept allows Google Analytics, Microsoft Clarity and PostHog, including session recording. Reject stops their recording. Cookieless counts and server request logs stay on. What we measure
Open data
55 small robots.txt cases, each with the result RFC 9309 gives and the sentence of the RFC it tests. Run them against any robots.txt parser to see where it reads the standard another way.
Updated
Each vector is a whole robots.txt file, a crawler name and a URL path. It says whether RFC 9309 allows that crawler to fetch that path, and which rule decides.
26 vectors come from 12 tests of answer-crawler-check, our open source robots.txt checker. They are its own cases, rewritten as data. 26 come from examples and sentences of the RFC itself. 3 come from the list of differences on our robots.txt sandbox.
We wrote each expected result from the RFC text, not from what our matcher returns. Every vector quotes the sentence of RFC 9309 it tests, word for word, with its section. We added a case from the RFC only where our matcher passes it, apart from the known differences. The cases it fails are listed under what the vectors leave out. The crawler names are placeholders such as ExampleBot, or the names the RFC's own examples use.
The file is at a stable address, and its version field says which release you have.
https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.jsonThe file is one JSON object, and its vectors array holds the cases. The fields around it name the version, the licence, the RFC text we read and the release of our tool.
| Field | Meaning |
|---|---|
id | A stable name for the vector. It never changes once published. |
robots_txt | The whole robots.txt file, as a string. Line ends are part of the test. |
user_agent | The product token of the crawler asking. |
path | The path and query of the URL to test, as the crawler would request it. |
expected | allowed or disallowed, as RFC 9309 reads the file. |
decided_by | The rule that decides the result: its line number in robots_txt, counted from 1, and its text without a comment. null when no rule decides: no rule matched, no group applied, or the path is /robots.txt. |
group | Which group the crawler obeys: own (a group that names it), * (the * group) or none. |
rfc | The section of RFC 9309 the vector tests, and the sentence it tests, word for word. level is the requirement keyword in that sentence, or null when it has none. |
reason | Why the result follows, in a sentence or two. |
from | Where the vector comes from. tool-test is a test of answer-crawler-check, named in test. rfc is an example or a sentence of RFC 9309. sandbox is the list of differences on the robots.txt sandbox page. |
matcher_differs | true when release 0.1.2 of the answer-crawler-check matcher gives the other result. |
difference | Present when matcher_differs is true. The id of the difference in the differences list. |
longest-match-allow{
"id": "longest-match-allow",
"robots_txt": "User-agent: *\nDisallow: /blog/\nAllow: /blog/public/\nDisallow: /tie\nAllow: /tie\n",
"user_agent": "ExampleBot",
"path": "/blog/public/post",
"expected": "allowed",
"decided_by": {
"line": 3,
"rule": "Allow: /blog/public/"
},
"group": "*",
"rfc": {
"section": "2.2.2",
"level": "MUST",
"quote": "The most specific match found MUST be used. The most specific match is the match that has the most octets."
},
"reason": "Allow: /blog/public/ and Disallow: /blog/ both match. The allow rule has more octets, so it decides.",
"from": {
"kind": "tool-test",
"test": "longest match wins and Allow wins a tie"
},
"matcher_differs": false
}Line numbers in decided_by count from 1. A carriage return, a line feed or the two together end a line, as the RFC's grammar says.
For each vector, give your parser the file, the crawler name and the path. Then compare its answer with expected. If your parser also reports the rule that decided, compare its line with decided_by.line.
This script runs them against answer-crawler-check. To test another parser, change the one line that calls it.
npm install answer-crawler-check@0.1.2// run-vectors.mjs, for Node 20 or later
import { allows, parseRobots } from 'answer-crawler-check'
const url = 'https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.json'
const { vectors } = await (await fetch(url)).json()
let failed = 0
for (const v of vectors) {
// Put your own parser here. It gets the file, the crawler name and the path.
const got = allows(parseRobots(v.robots_txt), v.user_agent, v.path).allowed ? 'allowed' : 'disallowed'
if (got !== v.expected) {
failed++
console.log(`${v.id}: expected ${v.expected}, got ${got}${v.matcher_differs ? ' (known difference)' : ''}`)
}
}
console.log(`${vectors.length - failed} of ${vectors.length} as expected`)
process.exitCode = failed ? 1 : 0With release 0.1.2 it prints 52 of 55 as expected and exits with code 1. The 3 it misses are the known differences below.
The same loop in Python, with your parser in is_allowed:
import json
import urllib.request
URL = "https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.json"
def is_allowed(robots_txt, user_agent, path):
raise NotImplementedError("call your parser here")
with urllib.request.urlopen(URL) as response:
vectors = json.load(response)["vectors"]
failed = []
for v in vectors:
got = "allowed" if is_allowed(v["robots_txt"], v["user_agent"], v["path"]) else "disallowed"
if got != v["expected"]:
failed.append(v["id"])
print(f"{len(vectors) - len(failed)} of {len(vectors)} as expected")
for vector_id in failed:
print("failed:", vector_id)rfc.level is the keyword in the quoted sentence, or null when the sentence has none. Where it is SHOULD, a parser may choose another way. That holds for 3 vectors: allow-wins-a-tie, path-case-is-kept, rule-before-any-group.
Once your parser runs the vectors in CI, a badge in your README can show its count. The script rfc9309-vectors-badge.mjs runs every vector through your parser and writes two files. One is an SVG badge, and the other is a JSON file for a shields.io badge. It needs Node 20 or later and nothing else.
Download the script into your repo.
curl -fsSL --create-dirs -o scripts/rfc9309-vectors-badge.mjs https://synapsereality.io/open-source/answer-crawler-check/rfc9309-vectors-badge.mjsMake isAllowed call your parser. It gets the whole file, the crawler name and the path, and returns true or false. Most parsers need an import at the top of the file and one line in the function. A parser in another language can run as a child process.
rfc9309-vectors-badge.mjs// Your parser goes here. It gets the whole robots.txt file, the crawler name
// and the path, and returns true when that crawler may fetch that path. It may
// be async. Replace the line inside, and import your parser at the top.
export function isAllowed(robotsTxt, userAgent, path) {
throw new Error('put your parser in isAllowed() in rfc9309-vectors-badge.mjs')
}Run it once, and commit the two files it writes.
node scripts/rfc9309-vectors-badge.mjs --out .github/badgesRun it in CI with --check, so a stale badge fails the build. Add --known with the ids of any vectors your parser reads another way on purpose.
node scripts/rfc9309-vectors-badge.mjs --out .github/badges --checkThe badge links back to this page. The second line is the credit the licence asks for.
[](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/)
Test cases from [RFC 9309 test vectors](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/) by Synapse, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).To draw it through shields.io instead, point an endpoint badge at the raw URL of the JSON file in your repo. Put your own names in place of OWNER and REPO, and your branch in place of main.
[](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/)--known takes the ids your parser reads another way on purpose. The run then passes, and the badge still counts them as misses. An error on one of them still fails the run.--check writes nothing. It fails when the files in your repo differ from a new run, so a stale badge fails the build. A badge that names an older patch release still passes if the count is the same.--check asks for a new badge. A new first number means a changed result or a vector taken out, and the script asks you to download it again.The file holds the robots.txt text, the quote and the reason for each row. A flagged row is one where our matcher gives the other result.
| Vector | Section | Crawler | Path | Expected | Decided by |
|---|---|---|---|---|---|
named-group-is-obeyed | 2.2.1 | ExampleBot | /page | allowed | line 5, Allow: / |
star-group-when-no-group-matches | 2.2.1 | OtherBot | /page | disallowed | line 2, Disallow: / |
named-group-takes-nothing-from-star | 2.2.2 | ExampleBot | /private/a | allowed | no rule |
product-token-any-case | 2.2.1 | ExampleBot | / | disallowed | line 2, Disallow: / |
sitemap-keeps-names-in-one-group-first | 2.2.4 | ExampleBot | /a | disallowed | line 4, Disallow: / |
sitemap-keeps-names-in-one-group-second | 2.2.4 | OtherBot | /a | disallowed | line 4, Disallow: / |
groups-for-one-name-merge-disallow | 2.2.1 | ExampleBot | /secret/x | disallowed | line 5, Disallow: /secret |
groups-for-one-name-merge-allow | 2.2.1 | ExampleBot | /open | allowed | line 2, Allow: / |
longest-match-disallow | 2.2.2 | ExampleBot | /blog/secret | disallowed | line 2, Disallow: /blog/ |
longest-match-allow | 2.2.2 | ExampleBot | /blog/public/post | allowed | line 3, Allow: /blog/public/ |
allow-wins-a-tie | 2.2.2 | ExampleBot | /tie | allowed | line 5, Allow: /tie |
dollar-ends-the-pattern | 2.2.3 | ExampleBot | /files/a.pdf | disallowed | line 2, Disallow: /*.pdf$ |
dollar-and-a-query | 2.2.3 | ExampleBot | /files/a.pdf?x=1 | allowed | no rule |
asterisk-then-prefix | 2.2.3 | ExampleBot | /a/b.html?x | disallowed | line 2, Disallow: /*.html |
slash-matches-every-path | 2.2.2 | ExampleBot | /anything | disallowed | line 2, Disallow: / |
dollar-blocks-a-longer-path | 2.2.3 | ExampleBot | /ab | allowed | no rule |
two-asterisks-and-dollar | 2.2.3 | ExampleBot | /axxbyyc | disallowed | line 2, Disallow: /a*b*c$ |
asterisk-matches-nothing | 2.2.3 | ExampleBot | / | disallowed | line 2, Disallow: /* |
empty-file | 2.2.1 | ExampleBot | /a | allowed | no rule |
group-with-no-rules | 2.1 | ExampleBot | /a | allowed | no rule |
robots-txt-is-allowed | 2.2.2 | ExampleBot | /robots.txt | allowed | no rule |
comment-after-a-rule | 2.2.3 | ExampleBot | /tmp/1 | disallowed | line 2, Disallow: /tmp |
comment-does-not-widen-a-rule | 2.2.3 | ExampleBot | /ok | allowed | no rule |
carriage-return-line-ends | 2.2 | ExampleBot | /tmp/1 | disallowed | line 2, Disallow: /tmp |
non-ascii-rule-and-encoded-path | 2.2.2 | ExampleBot | /%C3%BCber/a | disallowed | line 2, Disallow: /über |
encoded-rule-and-non-ascii-path | 2.2.2 | ExampleBot | /€ | disallowed | line 2, Disallow: /%E2%82%AC |
empty-line-inside-a-group | 2.2 | ExampleBot | /admin/a | disallowed | line 3, Disallow: /admin |
path-case-is-kept | 2.2.2 | ExampleBot | /blog | allowed | no rule |
match-starts-at-the-first-octet | 2.2.2 | ExampleBot | /public/private | allowed | no rule |
rule-before-any-group | 2.2.2 | ExampleBot | /page | allowed | no rule |
user-agent-line-starts-a-new-group | 2.1 | ExampleBot | /b | allowed | no rule |
no-group-and-no-star | 2.2.1 | ExampleBot | /x | allowed | no rule |
spaces-and-tabs-around-the-key | 2.2 | ExampleBot | /tmp/x | disallowed | line 2, disallow : /tmp |
merge-figure-2 | 2.2.1 | ExampleBot | /baz | disallowed | line 6, disallow: /baz |
star-figure-3-rule | 2.2.1 | ExampleBot | /foo | disallowed | line 2, disallow: /foo |
star-figure-3-other-group | 2.2.1 | ExampleBot | /baz | allowed | no rule |
encoded-utf8-figure-4 | 2.2.2 | ExampleBot | /foo/bar/ツ | disallowed | line 2, Disallow: /foo/bar/%E3%83%84 |
dollar-figure-5-exact | 2.2.3 | ExampleBot | /this/path/exactly | allowed | line 3, Allow: /this/path/exactly$ |
dollar-figure-5-longer | 2.2.3 | ExampleBot | /this/path/exactly/more | disallowed | line 2, Disallow: / |
asterisk-figure-5 | 2.2.3 | ExampleBot | /this/is/exactly | allowed | line 3, Allow: /this/*/exactly |
example-5-1-star-gif | 5.1 | ExampleBot | /images/logo.gif | disallowed | line 2, Disallow: *.gif$ |
example-5-1-star-example | 5.1 | ExampleBot | /example/x.html | disallowed | line 3, Disallow: /example/ |
example-5-1-star-publications | 5.1 | ExampleBot | /publications/report.html | allowed | line 4, Allow: /publications/ |
example-5-1-foobot-page | 5.1 | foobot | /example/page.html | allowed | line 8, Allow:/example/page.html |
example-5-1-foobot-gif | 5.1 | foobot | /example/allowed.gif | allowed | line 9, Allow:/example/allowed.gif |
example-5-1-foobot-other | 5.1 | foobot | /index.html | disallowed | line 7, Disallow:/ |
example-5-1-barbot-page | 5.1 | barbot | /example/page.html | disallowed | line 13, Disallow: /example/page.html |
example-5-1-bazbot-page | 5.1 | bazbot | /example/page.html | disallowed | line 13, Disallow: /example/page.html |
example-5-1-barbot-other | 5.1 | barbot | /example/other.html | allowed | no rule |
example-5-1-quxbot | 5.1 | quxbot | /example/page.html | allowed | no rule |
example-5-2-longest | 5.2 | foobot | /example/page/disallowed.gif | disallowed | line 3, Disallow: /example/page/disallowed.gif |
other-record-keeps-names-in-one-group-second | 2.2.4 | OtherBot | /x | disallowed | line 4, Disallow: / |
other-record-keeps-names-in-one-group-first flagged | 2.2.4 | ExampleBot | /x | disallowed | line 4, Disallow: / |
unreserved-escape-in-the-path flagged | 2.2.2 | ExampleBot | /%70rivate | disallowed | line 2, Disallow: /private |
unreserved-escape-figure-4 flagged | 2.2.2 | ExampleBot | /foo/bar/%62%61%7A | disallowed | line 2, Disallow: /foo/bar/baz |
A review of release 0.1.2 of the answer-crawler-check matcher found these 2 differences from the RFC. The vectors for them give the RFC's result and carry "matcher_differs": true. Two more cases it fails are under what the vectors leave out.
/private does not match the path /%70rivate, and RFC 9309 says it should. Section 2.2.2. Vectors: unreserved-escape-in-the-path, unreserved-escape-figure-4.Crawl-delay. The names above that line get none of the rules below it. RFC 9309 keeps them in one group. Section 2.2.4. Vectors: other-record-keeps-names-in-one-group-first.Our site build runs every vector against the copy of the matcher it uses. The build fails if any other vector stops passing, and also if a flagged one starts to pass. A fixed release takes its flags off, with a new version of this file.
Disallow: line. Read word for word, the matching rule of section 2.2.2 matches it to every path. The 1994 page the RFC cites says an empty value allows every URL. The two readings disagree, so there is no vector for it.ExampleBot/1.0 on a user-agent line. The RFC does not describe either.%2A for a literal *. Release 0.1.2 does not match them, so they are not vectors.https://foo.bar in a query. Release 0.1.2 does not percent-encode reserved characters in a rule, so that row is not a vector either.The vectors are under CC BY 4.0, the licence of our other open data. You may copy them into any project and change them, open source or not. In return, credit us, link the licence and say if you changed them.
For us, a comment like this in your test file is enough, with a line added if you changed the vectors:
// Test cases from RFC 9309 test vectors 1.0.0 by Synapse
// https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/
// Licence: CC BY 4.0, https://creativecommons.org/licenses/by/4.0/Text taken from RFC 9309 is not ours. That is the quotes, and the robots.txt files copied from the RFC's figures and examples. The RFC is the IETF Trust's, and the licence above does not cover that text.
A vector's id never changes. A new vector raises the middle number of the version. A changed expected result, or a vector taken out, raises the first. A fix to a reason or a quote raises the last.