SynapseContact us

// open source

answer-crawler-check

Code
github.com/synapsereality/answer-crawler-check
Install
npx answer-crawler-check
Runs on
Node.js 20 or later
Licence
MIT, free to use

answer-crawler-check fails your build when robots.txt blocks an AI answer crawler from any page in your sitemap.

npx answer-crawler-check https://example.com/sitemap-index.xml

Why it exists

Each AI engine reads robots.txt through its own crawler names. OpenAI documents OAI-SearchBot for search and GPTBot for training, each with its own robots.txt entry. Block the first by mistake and ChatGPT search can’t fetch the page it would have cited.

We saw the gap on this site. In a review of the build, someone added User-agent: Claude-SearchBot and Disallow: / to the built file, and every other check passed. Since then the build tests every page for every answer crawler we allow. This tool is that test, for any site.

A check of your home page, or of one URL you paste in, misses a rule like Disallow: /docs/ in one crawler’s group, or /*.html in another. Those block some pages and leave the home page open. This tool reads every URL in the sitemap.

What you get back

https://example.com: robots.txt from dist/robots.txt, local file, 3 page(s)
ok      OAI-SearchBot  (no group of its own, uses *)
BLOCKED Claude-SearchBot  1 URL(s)
          https://example.com/docs/setup/  Disallow: /docs/ (line 9)
...
answer-crawler-check: FAILED, 2 problem(s)

Each blocked page comes with the rule and line that blocked it.

Which crawlers it checks

By default it checks 11 crawlers. They fetch pages while an answer is written, or build the index the answers cite. They are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Applebot, Googlebot and Bingbot.

Training crawlers such as GPTBot and CCBot are left out. Blocking them is a fair choice, and it doesn’t take you out of search answers. Name them with --agents if you want them checked too.

The crawler names come from ai.robots.txt, the community list of AI crawlers, used under its MIT licence. A copy ships with the tool, and --list latest fetches the current one.

Check before you deploy

npm run build
npx answer-crawler-check dist/sitemap-index.xml --site-dir dist

With --site-dir, the sitemaps and robots.txt come from your build folder, so a bad rule fails the build before it reaches the live site.

Running it in CI

Every job below fails when a crawler is blocked. The ones that check a build assume npm run build writes the site to dist/.

GitHub Actions

- uses: synapsereality/answer-crawler-check@v0.1.1
  with:
    sitemap: dist/sitemap-index.xml
    site-dir: dist

GitLab CI

In .gitlab-ci.yml:

answer-crawlers:
  image: node:22-slim
  script:
    - npm ci
    - npm run build
    - npx --yes answer-crawler-check@0.1.1 dist/sitemap-index.xml --site-dir dist --json crawlers.json
  artifacts:
    when: always
    paths:
      - crawlers.json

when: always keeps the JSON report when the job fails. To check the live site on a schedule, add a job that only runs when $CI_PIPELINE_SOURCE is "schedule", and create the schedule under Build > Pipeline schedules.

Bitbucket Pipelines

In bitbucket-pipelines.yml:

image: node:22-slim

pipelines:
  default:
    - step:
        name: Check AI answer crawler access
        script:
          - npm ci
          - npm run build
          - npx --yes answer-crawler-check@0.1.1 dist/sitemap-index.xml --site-dir dist

For a weekly check of the live site, put the same npx line with your live sitemap URL in a custom pipeline. Then schedule it under Pipelines, Schedules.

Any other CI

Anything that runs a shell command and reads its exit code works, with Node 20 or later:

npm run build
npx --yes answer-crawler-check@0.1.1 dist/sitemap-index.xml --site-dir dist

Exit codes

Code Meaning
0 Every crawler can fetch every URL.
1 A crawler is blocked, a sitemap or robots file couldn’t be read, or there were no URLs to check. The report lists each problem. A crash also gives 1.
2 The command line was wrong, so nothing was checked.

The README has the scheduled jobs in full and a shell step that tells codes 1 and 2 apart.

What it can’t see

robots.txt is one of several ways to turn a crawler away. A firewall rule, a bot challenge or a noindex header can block a crawler this tool calls allowed. Our free instant audit reads the HTML a crawler receives for one page.

< All open source