Open data

RFC 9309 test vectors for robots.txt parsers

55 small robots.txt cases, each with the result RFC 9309 gives and the sentence of the RFC it tests. Run them against any robots.txt parser to see where it reads the standard another way.

Download the JSON 46 KB

Updated

What the file is

Each vector is a whole robots.txt file, a crawler name and a URL path. It says whether RFC 9309 allows that crawler to fetch that path, and which rule decides.

26 vectors come from 12 tests of answer-crawler-check, our open source robots.txt checker. They are its own cases, rewritten as data. 26 come from examples and sentences of the RFC itself. 3 come from the list of differences on our robots.txt sandbox.

We wrote each expected result from the RFC text, not from what our matcher returns. Every vector quotes the sentence of RFC 9309 it tests, word for word, with its section. We added a case from the RFC only where our matcher passes it, apart from the known differences. The cases it fails are listed under what the vectors leave out. The crawler names are placeholders such as ExampleBot, or the names the RFC's own examples use.

The file is at a stable address, and its version field says which release you have.

https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.json

What a vector holds

The file is one JSON object, and its vectors array holds the cases. The fields around it name the version, the licence, the RFC text we read and the release of our tool.

The fields of each vector, as the file's own fields object describes them
FieldMeaning
idA stable name for the vector. It never changes once published.
robots_txtThe whole robots.txt file, as a string. Line ends are part of the test.
user_agentThe product token of the crawler asking.
pathThe path and query of the URL to test, as the crawler would request it.
expectedallowed or disallowed, as RFC 9309 reads the file.
decided_byThe rule that decides the result: its line number in robots_txt, counted from 1, and its text without a comment. null when no rule decides: no rule matched, no group applied, or the path is /robots.txt.
groupWhich group the crawler obeys: own (a group that names it), * (the * group) or none.
rfcThe section of RFC 9309 the vector tests, and the sentence it tests, word for word. level is the requirement keyword in that sentence, or null when it has none.
reasonWhy the result follows, in a sentence or two.
fromWhere the vector comes from. tool-test is a test of answer-crawler-check, named in test. rfc is an example or a sentence of RFC 9309. sandbox is the list of differences on the robots.txt sandbox page.
matcher_differstrue when release 0.1.2 of the answer-crawler-check matcher gives the other result.
differencePresent when matcher_differs is true. The id of the difference in the differences list.
One vector, longest-match-allow
{
  "id": "longest-match-allow",
  "robots_txt": "User-agent: *\nDisallow: /blog/\nAllow: /blog/public/\nDisallow: /tie\nAllow: /tie\n",
  "user_agent": "ExampleBot",
  "path": "/blog/public/post",
  "expected": "allowed",
  "decided_by": {
    "line": 3,
    "rule": "Allow: /blog/public/"
  },
  "group": "*",
  "rfc": {
    "section": "2.2.2",
    "level": "MUST",
    "quote": "The most specific match found MUST be used. The most specific match is the match that has the most octets."
  },
  "reason": "Allow: /blog/public/ and Disallow: /blog/ both match. The allow rule has more octets, so it decides.",
  "from": {
    "kind": "tool-test",
    "test": "longest match wins and Allow wins a tie"
  },
  "matcher_differs": false
}

Line numbers in decided_by count from 1. A carriage return, a line feed or the two together end a line, as the RFC's grammar says.

Run them against your parser

For each vector, give your parser the file, the crawler name and the path. Then compare its answer with expected. If your parser also reports the rule that decided, compare its line with decided_by.line.

This script runs them against answer-crawler-check. To test another parser, change the one line that calls it.

JavaScript, with npm install answer-crawler-check@0.1.2
// run-vectors.mjs, for Node 20 or later
import { allows, parseRobots } from 'answer-crawler-check'

const url = 'https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.json'
const { vectors } = await (await fetch(url)).json()
let failed = 0
for (const v of vectors) {
  // Put your own parser here. It gets the file, the crawler name and the path.
  const got = allows(parseRobots(v.robots_txt), v.user_agent, v.path).allowed ? 'allowed' : 'disallowed'
  if (got !== v.expected) {
    failed++
    console.log(`${v.id}: expected ${v.expected}, got ${got}${v.matcher_differs ? ' (known difference)' : ''}`)
  }
}
console.log(`${vectors.length - failed} of ${vectors.length} as expected`)
process.exitCode = failed ? 1 : 0

With release 0.1.2 it prints 52 of 55 as expected and exits with code 1. The 3 it misses are the known differences below.

The same loop in Python, with your parser in is_allowed:

Python 3, standard library only
import json
import urllib.request

URL = "https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors.json"


def is_allowed(robots_txt, user_agent, path):
    raise NotImplementedError("call your parser here")


with urllib.request.urlopen(URL) as response:
    vectors = json.load(response)["vectors"]

failed = []
for v in vectors:
    got = "allowed" if is_allowed(v["robots_txt"], v["user_agent"], v["path"]) else "disallowed"
    if got != v["expected"]:
        failed.append(v["id"])

print(f"{len(vectors) - len(failed)} of {len(vectors)} as expected")
for vector_id in failed:
    print("failed:", vector_id)

rfc.level is the keyword in the quoted sentence, or null when the sentence has none. Where it is SHOULD, a parser may choose another way. That holds for 3 vectors: allow-wins-a-tie, path-case-is-kept, rule-before-any-group.

Show your result

Once your parser runs the vectors in CI, a badge in your README can show its count. The script rfc9309-vectors-badge.mjs runs every vector through your parser and writes two files. One is an SVG badge, and the other is a JSON file for a shields.io badge. It needs Node 20 or later and nothing else.

RFC 9309 vectors v1.0.0: 52 of 55
answer-crawler-check's result, drawn by this script. Its CI runs the script on every push, and its README shows the badge.
  1. Download the script into your repo.

    Shell
    curl -fsSL --create-dirs -o scripts/rfc9309-vectors-badge.mjs https://synapsereality.io/open-source/answer-crawler-check/rfc9309-vectors-badge.mjs
  2. Make isAllowed call your parser. It gets the whole file, the crawler name and the path, and returns true or false. Most parsers need an import at the top of the file and one line in the function. A parser in another language can run as a child process.

    The function to change, in rfc9309-vectors-badge.mjs
    // Your parser goes here. It gets the whole robots.txt file, the crawler name
    // and the path, and returns true when that crawler may fetch that path. It may
    // be async. Replace the line inside, and import your parser at the top.
    export function isAllowed(robotsTxt, userAgent, path) {
      throw new Error('put your parser in isAllowed() in rfc9309-vectors-badge.mjs')
    }
  3. Run it once, and commit the two files it writes.

    Shell
    node scripts/rfc9309-vectors-badge.mjs --out .github/badges
  4. Run it in CI with --check, so a stale badge fails the build. Add --known with the ids of any vectors your parser reads another way on purpose.

    Shell, in CI
    node scripts/rfc9309-vectors-badge.mjs --out .github/badges --check

The README snippet

The badge links back to this page. The second line is the credit the licence asks for.

Markdown
[![RFC 9309 test vectors result](.github/badges/rfc9309-vectors-badge.svg)](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/)

Test cases from [RFC 9309 test vectors](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/) by Synapse, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

To draw it through shields.io instead, point an endpoint badge at the raw URL of the JSON file in your repo. Put your own names in place of OWNER and REPO, and your branch in place of main.

Markdown, through shields.io
[![RFC 9309 test vectors result](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/OWNER/REPO/main/.github/badges/rfc9309-vectors-badge.json)](https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/)

What the badge says

  • How many vectors gave the expected result, and the version of the file. It is a count and not a verdict. A parser can read the 3 vectors that test a SHOULD another way and still follow the RFC.
  • --known takes the ids your parser reads another way on purpose. The run then passes, and the badge still counts them as misses. An error on one of them still fails the run.
  • --check writes nothing. It fails when the files in your repo differ from a new run, so a stale badge fails the build. A badge that names an older patch release still passes if the count is the same.
  • The script fetches the vectors file from this site and calls nothing else. The badge is a file in your own repo, so a view of your README never reaches us.
  • The label names the version of the vectors it ran. A new middle number adds vectors, so the count changes and --check asks for a new badge. A new first number means a changed result or a vector taken out, and the script asks you to download it again.

What the vectors cover

Vectors by the section of RFC 9309 they quote
SectionTitle in the RFCVectorsFlagged
2.1Protocol Definition20
2.2Formal Syntax30
2.2.1The User-Agent Line100
2.2.2The "Allow" and "Disallow" Lines142
2.2.3Special Characters110
2.2.4Other Records41
5.1Simple Example100
5.2Longest Match10

Every vector

The file holds the robots.txt text, the quote and the reason for each row. A flagged row is one where our matcher gives the other result.

All 55 vectors, in the order of the file
VectorSectionCrawlerPathExpectedDecided by
named-group-is-obeyed2.2.1ExampleBot/pageallowedline 5, Allow: /
star-group-when-no-group-matches2.2.1OtherBot/pagedisallowedline 2, Disallow: /
named-group-takes-nothing-from-star2.2.2ExampleBot/private/aallowedno rule
product-token-any-case2.2.1ExampleBot/disallowedline 2, Disallow: /
sitemap-keeps-names-in-one-group-first2.2.4ExampleBot/adisallowedline 4, Disallow: /
sitemap-keeps-names-in-one-group-second2.2.4OtherBot/adisallowedline 4, Disallow: /
groups-for-one-name-merge-disallow2.2.1ExampleBot/secret/xdisallowedline 5, Disallow: /secret
groups-for-one-name-merge-allow2.2.1ExampleBot/openallowedline 2, Allow: /
longest-match-disallow2.2.2ExampleBot/blog/secretdisallowedline 2, Disallow: /blog/
longest-match-allow2.2.2ExampleBot/blog/public/postallowedline 3, Allow: /blog/public/
allow-wins-a-tie2.2.2ExampleBot/tieallowedline 5, Allow: /tie
dollar-ends-the-pattern2.2.3ExampleBot/files/a.pdfdisallowedline 2, Disallow: /*.pdf$
dollar-and-a-query2.2.3ExampleBot/files/a.pdf?x=1allowedno rule
asterisk-then-prefix2.2.3ExampleBot/a/b.html?xdisallowedline 2, Disallow: /*.html
slash-matches-every-path2.2.2ExampleBot/anythingdisallowedline 2, Disallow: /
dollar-blocks-a-longer-path2.2.3ExampleBot/aballowedno rule
two-asterisks-and-dollar2.2.3ExampleBot/axxbyycdisallowedline 2, Disallow: /a*b*c$
asterisk-matches-nothing2.2.3ExampleBot/disallowedline 2, Disallow: /*
empty-file2.2.1ExampleBot/aallowedno rule
group-with-no-rules2.1ExampleBot/aallowedno rule
robots-txt-is-allowed2.2.2ExampleBot/robots.txtallowedno rule
comment-after-a-rule2.2.3ExampleBot/tmp/1disallowedline 2, Disallow: /tmp
comment-does-not-widen-a-rule2.2.3ExampleBot/okallowedno rule
carriage-return-line-ends2.2ExampleBot/tmp/1disallowedline 2, Disallow: /tmp
non-ascii-rule-and-encoded-path2.2.2ExampleBot/%C3%BCber/adisallowedline 2, Disallow: /über
encoded-rule-and-non-ascii-path2.2.2ExampleBot/€disallowedline 2, Disallow: /%E2%82%AC
empty-line-inside-a-group2.2ExampleBot/admin/adisallowedline 3, Disallow: /admin
path-case-is-kept2.2.2ExampleBot/blogallowedno rule
match-starts-at-the-first-octet2.2.2ExampleBot/public/privateallowedno rule
rule-before-any-group2.2.2ExampleBot/pageallowedno rule
user-agent-line-starts-a-new-group2.1ExampleBot/ballowedno rule
no-group-and-no-star2.2.1ExampleBot/xallowedno rule
spaces-and-tabs-around-the-key2.2ExampleBot/tmp/xdisallowedline 2, disallow : /tmp
merge-figure-22.2.1ExampleBot/bazdisallowedline 6, disallow: /baz
star-figure-3-rule2.2.1ExampleBot/foodisallowedline 2, disallow: /foo
star-figure-3-other-group2.2.1ExampleBot/bazallowedno rule
encoded-utf8-figure-42.2.2ExampleBot/foo/bar/ツdisallowedline 2, Disallow: /foo/bar/%E3%83%84
dollar-figure-5-exact2.2.3ExampleBot/this/path/exactlyallowedline 3, Allow: /this/path/exactly$
dollar-figure-5-longer2.2.3ExampleBot/this/path/exactly/moredisallowedline 2, Disallow: /
asterisk-figure-52.2.3ExampleBot/this/is/exactlyallowedline 3, Allow: /this/*/exactly
example-5-1-star-gif5.1ExampleBot/images/logo.gifdisallowedline 2, Disallow: *.gif$
example-5-1-star-example5.1ExampleBot/example/x.htmldisallowedline 3, Disallow: /example/
example-5-1-star-publications5.1ExampleBot/publications/report.htmlallowedline 4, Allow: /publications/
example-5-1-foobot-page5.1foobot/example/page.htmlallowedline 8, Allow:/example/page.html
example-5-1-foobot-gif5.1foobot/example/allowed.gifallowedline 9, Allow:/example/allowed.gif
example-5-1-foobot-other5.1foobot/index.htmldisallowedline 7, Disallow:/
example-5-1-barbot-page5.1barbot/example/page.htmldisallowedline 13, Disallow: /example/page.html
example-5-1-bazbot-page5.1bazbot/example/page.htmldisallowedline 13, Disallow: /example/page.html
example-5-1-barbot-other5.1barbot/example/other.htmlallowedno rule
example-5-1-quxbot5.1quxbot/example/page.htmlallowedno rule
example-5-2-longest5.2foobot/example/page/disallowed.gifdisallowedline 3, Disallow: /example/page/disallowed.gif
other-record-keeps-names-in-one-group-second2.2.4OtherBot/xdisallowedline 4, Disallow: /
other-record-keeps-names-in-one-group-first flagged2.2.4ExampleBot/xdisallowedline 4, Disallow: /
unreserved-escape-in-the-path flagged2.2.2ExampleBot/%70rivatedisallowedline 2, Disallow: /private
unreserved-escape-figure-4 flagged2.2.2ExampleBot/foo/bar/%62%61%7Adisallowedline 2, Disallow: /foo/bar/baz

Where our matcher differs

A review of release 0.1.2 of the answer-crawler-check matcher found these 2 differences from the RFC. The vectors for them give the RFC's result and carry "matcher_differs": true. Two more cases it fails are under what the vectors leave out.

  • The matcher compares a percent-encoded unreserved character as written. A rule for /private does not match the path /%70rivate, and RFC 9309 says it should. Section 2.2.2. Vectors: unreserved-escape-in-the-path, unreserved-escape-figure-4.
  • The matcher ends a run of user-agent lines at a line that is not a rule, such as Crawl-delay. The names above that line get none of the rules below it. RFC 9309 keeps them in one group. Section 2.2.4. Vectors: other-record-keeps-names-in-one-group-first.

Our site build runs every vector against the copy of the matcher it uses. The build fails if any other vector stops passing, and also if a flagged one starts to pass. A fixed release takes its flags off, with a new version of this file.

What the vectors leave out

  • Fetching the file. Section 2.3 covers redirects, status codes and an unreachable server, and section 2.4 covers caching. A vector here starts from the text of the file.
  • The parsing limit of at least 500 KiB in section 2.5. A vector for it would need a file larger than that.
  • An empty Disallow: line. Read word for word, the matching rule of section 2.2.2 matches it to every path. The 1994 page the RFC cites says an empty value allows every URL. The two readings disagree, so there is no vector for it.
  • A byte order mark, and a version after a name, as in ExampleBot/1.0 on a user-agent line. The RFC does not describe either.
  • The escaped special characters of Figure 6, such as %2A for a literal *. Release 0.1.2 does not match them, so they are not vectors.
  • The row of Figure 4 with https://foo.bar in a query. Release 0.1.2 does not percent-encode reserved characters in a rule, so that row is not a vector either.

Licence and credit

The vectors are under CC BY 4.0, the licence of our other open data. You may copy them into any project and change them, open source or not. In return, credit us, link the licence and say if you changed them.

For us, a comment like this in your test file is enough, with a line added if you changed the vectors:

// Test cases from RFC 9309 test vectors 1.0.0 by Synapse
// https://synapsereality.io/open-source/answer-crawler-check/rfc9309-test-vectors/
// Licence: CC BY 4.0, https://creativecommons.org/licenses/by/4.0/

Text taken from RFC 9309 is not ours. That is the quotes, and the robots.txt files copied from the RFC's figures and examples. The RFC is the IETF Trust's, and the licence above does not cover that text.

Versions

A vector's id never changes. A new vector raises the middle number of the version. A changed expected result, or a vector taken out, raises the first. A fix to a reason or a quote raises the last.

  • 1.0.0, : the first release, with 55 vectors. Each RFC quote was checked against the text of RFC 9309 we read from the RFC Editor that day.