Measure this visit?
Accept allows Google Analytics, Microsoft Clarity and PostHog, including session recording. Reject stops their recording. Cookieless counts and server request logs stay on. What we measure
Tool
Paste raw HTTP response headers, such as the output of curl -I. The reader picks out the status line, the redirect chain, X-Robots-Tag, the canonical link, Content-Type, Retry-After and a bot challenge marker. Each result names the line that decided it and links its source. It runs in your browser. Nothing you paste is sent, fetched or stored on a server.
The reader reads the text you paste. It does not show what any crawler receives. To fetch a page and read what came back, use the instant audit. To test robots.txt rules, use the robots.txt sandbox.
Updated
The output of curl -I works, and so do several responses in a row. Take out any cookie or token first if you like. Nothing leaves this page either way.
Paste headers or load an example. The reading shows here as you type.
From a terminal, ask for the headers only and follow redirects. Each response in the chain prints its own block, and you can paste them all at once.
curl -sIL https://example.com/pageThe verbose output of curl -v works too. So does a block of headers copied from a browser's network panel, with or without a status line. A paste can hold up to 100,000 characters.
The reader explains a line only when it holds a source for it. A source is an HTTP standard or the vendor's own page. Each statement below is ours, and under it is the sentence it rests on, as written on the linked page. We read every page on 14 Nov 2026. A header, a status code or a rule with no row here gets no explanation.
If the server responded with a 2xx status code, the content received in the response may be considered for indexing.Google: How HTTP status codes affect Google's crawlers
For Google Search, an HTTP 2xx (success) status code doesn't guarantee indexing.Google: How HTTP status codes affect Google's crawlers
The 301 (Moved Permanently) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.RFC 9110: HTTP Semantics
Google follows the redirect, and Google systems use the redirect as a strong signal that the redirect target should be processed.Google: How HTTP status codes affect Google's crawlers
The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.RFC 9110: HTTP Semantics
Equivalent to 301.Google: How HTTP status codes affect Google's crawlers
The 302 (Found) status code indicates that the target resource resides temporarily under a different URI.RFC 9110: HTTP Semantics
By default, Google's crawlers follow the redirect, and Google systems use the redirect as a weak signal that the redirect target should be processed.Google: How HTTP status codes affect Google's crawlers
The 307 (Temporary Redirect) status code indicates that the target resource resides temporarily under a different URI and the user agent MUST NOT change the request method if it performs an automatic redirection to that URI.RFC 9110: HTTP Semantics
Equivalent to 302.Google: How HTTP status codes affect Google's crawlers
The 303 (See Other) status code indicates that the server is redirecting the user agent to a different resource, as indicated by a URI in the Location header field, which is intended to provide an indirect response to the original request.RFC 9110: HTTP Semantics
By default, Google's crawlers follow the redirect, and Google systems use the redirect as a weak signal that the redirect target should be processed.Google: How HTTP status codes affect Google's crawlers
Any content Google receives from the redirecting URL is ignored, and the final target URL's content is processed instead.Google: How HTTP status codes affect Google's crawlers
Google doesn't use the content from URLs that return 4xx status codes.Google: How HTTP status codes affect Google's crawlers
In the case of Google Search, Google doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index.Google: How HTTP status codes affect Google's crawlers
The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting").RFC 6585: Additional HTTP Status Codes
Google's crawlers treat the 429 status code as a signal that the server is overloaded, and it's considered a server error.Google: How HTTP status codes affect Google's crawlers
The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.RFC 9110: HTTP Semantics
5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling.Google: How HTTP status codes affect Google's crawlers
For Google Search, already indexed URLs are preserved in the index, but eventually dropped.Google: How HTTP status codes affect Google's crawlers
By default, Google's crawlers follow up to 10 redirect hops.Google: How HTTP status codes affect Google's crawlers
For 3xx (Redirection) responses, the Location value refers to the preferred target resource for automatically redirecting the request.RFC 9110: HTTP Semantics
When it has the form of a relative reference ([URI], Section 4.2), the final value is computed by resolving it against the target URI ([URI], Section 5).RFC 9110: HTTP Semantics
Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The X-Robots-Tag may optionally specify a user agent before the rules.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Rules specified without a user agent are valid for all crawlers.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
In the case of conflicting robots rules, the more restrictive rule applies.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
It is possible that these rules may not be treated the same by all other search engines.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Do not show this page, media, or resource in search results.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Do not follow the links on this page.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Equivalent to noindex, nofollow.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Do not show a text snippet or video preview in the search results for this page.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
indexifembedded only has an effect if it's accompanied by noindex.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Use a maximum of [number] characters as a textual snippet for this search result.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Set the maximum size of an image preview for this page in search results.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Use a maximum of [number] seconds as a video snippet for videos on this page in search results.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
No snippet is to be shown.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google will choose the snippet length that it believes is most effective to help users discover your content and direct users to your site.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
No image preview is to be shown.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
A default image preview may be shown.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
A larger image preview, up to the width of the viewport, may be shown.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
At most, a static image may be used, in accordance to the max-image-preview setting.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
There is no limit.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Don't offer translation of this page in search results.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Do not index images on this page.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Do not show this page in search results after the specified date/time.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The nocache rule isn't used by Google Search.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The nositelinkssearchbox rule is no longer used by Google Search to control whether the sitelink search box is shown for a given page, as the feature no longer exists.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The Link header field provides a means for serialising one or more links into HTTP headers.RFC 8288: Web Linking
The canonical link relation specifies the preferred IRI from resources with duplicative content.RFC 6596: The Canonical Link Relation
Google supports this method for web search results only.Google: How to specify a canonical URL with rel="canonical" and other methods
As with the rel="canonical" link element, use absolute URLs in the rel="canonical" HTTP header.Google: How to specify a canonical URL with rel="canonical" and other methods
The "Content-Type" header field indicates the media type of the associated representation: either the representation enclosed in the message content or the selected representation, as determined by the message semantics.RFC 9110: HTTP Semantics
To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Servers send the "Retry-After" header field to indicate how long the user agent ought to wait before making a follow-up request.RFC 9110: HTTP Semantics
The Retry-After field value can be either an HTTP-date or a number of seconds to delay after receiving the response.RFC 9110: HTTP Semantics
When sent with a 503 (Service Unavailable) response, Retry-After indicates how long the service is expected to be unavailable to the client.RFC 9110: HTTP Semantics
The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.RFC 6585: Additional HTTP Status Codes
When sent with any 3xx (Redirection) response, Retry-After indicates the minimum time that the user agent is asked to wait before issuing the redirected request.RFC 9110: HTTP Semantics
When a request encounters a Cloudflare Challenge Page instead of the originally anticipated response, the Challenge Page response (regardless of the Challenge Page type) will have the cf-mitigated header present and set to challenge.Cloudflare: Detect a Challenge Page response
Regardless of the requested resource-type, the content-type of a challenge will be text/html.Cloudflare: Detect a Challenge Page response
For the cf-mitigated header, challenge is the only valid value.Cloudflare: Detect a Challenge Page response
The sentences about X-Robots-Tag above are Google's, from its page for Google Search. Google says the X-Robots-Tag header may name a user agent before the rules. Google says rules with no user agent in front are valid for all crawlers.
When a response has an X-Robots-Tag line, the reader lists the rules it addresses to each name in our AI crawler tokens table: ClaudeBot, Claude-SearchBot, Claude-User, GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, Google-Extended, PerplexityBot, Perplexity-User. It shows what is written in the header. It does not say that any of those crawlers obeys it.
The names come from the crawler pages of Anthropic, OpenAI, Google and Perplexity. On 14 Nov 2026 we searched each of those pages for "X-Robots-Tag", "noindex", "Retry-After" and "cf-mitigated" and found no match. So the reader says of each name only that its vendor's page does not mention the header.
It cannot tell you what a crawler does. It reads the lines you paste and repeats what a standard or a vendor's page says about them. A crawler may act another way.
It does not fetch anything. A server can answer one client differently from another, so the headers you paste are the ones your request got. It cannot see the page itself, so it misses a robots meta tag or a canonical link in the HTML. The instant audit fetches one page from our server and reads its HTML.
It does not read robots.txt. The robots.txt sandbox tests a path against the rules of a pasted file, for each AI crawler name.
It does not know the address you asked for. It lists each Location value as written and does not check that the next response came from it. To check a whole redirect map from a terminal, use one-hop.
It stays in your browser. The reader makes no request with it, and closing or reloading the tab discards it. The session recording and the click analytics on the rest of this site read text off a page, so this page does not load them.
There is no link to share a reading. Response headers often hold a cookie or a token, and a link would carry it.