Tool

HTTP response header reader: the lines that decide crawling and indexing

Paste raw HTTP response headers, such as the output of curl -I. The reader picks out the status line, the redirect chain, X-Robots-Tag, the canonical link, Content-Type, Retry-After and a bot challenge marker. Each result names the line that decided it and links its source. It runs in your browser. Nothing you paste is sent, fetched or stored on a server.

The reader reads the text you paste. It does not show what any crawler receives. To fetch a page and read what came back, use the instant audit. To test robots.txt rules, use the robots.txt sandbox.

Updated

The headers

The output of curl -I works, and so do several responses in a row. Take out any cookie or token first if you like. Nothing leaves this page either way.

What the headers say

Paste headers or load an example. The reading shows here as you type.

How to get the headers with curl

From a terminal, ask for the headers only and follow redirects. Each response in the chain prints its own block, and you can paste them all at once.

curl -sIL https://example.com/page

The verbose output of curl -v works too. So does a block of headers copied from a browser's network panel, with or without a status line. A paste can hold up to 100,000 characters.

What each line means, and who says so

The reader explains a line only when it holds a source for it. A source is an HTTP standard or the vendor's own page. Each statement below is ours, and under it is the sentence it rests on, as written on the linked page. We read every page on 14 Nov 2026. A header, a status code or a rule with no row here gets no explanation.

The status line

Google says the content of a 2xx response may be considered for indexing.
If the server responded with a 2xx status code, the content received in the response may be considered for indexing. Google: How HTTP status codes affect Google's crawlers
Google says that for Google Search a 2xx status does not guarantee indexing.
For Google Search, an HTTP 2xx (success) status code doesn't guarantee indexing. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 301 means the resource has a new permanent address.
The 301 (Moved Permanently) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs. RFC 9110: HTTP Semantics
Google says it follows a 301 and uses it as a strong signal to process the target.
Google follows the redirect, and Google systems use the redirect as a strong signal that the redirect target should be processed. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 308 means the resource has a new permanent address.
The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs. RFC 9110: HTTP Semantics
Google says it treats a 308 as it treats a 301.
Equivalent to 301. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 302 means the resource is at another address for now.
The 302 (Found) status code indicates that the target resource resides temporarily under a different URI. RFC 9110: HTTP Semantics
Google says its crawlers follow a 302 by default and use it as a weak signal to process the target.
By default, Google's crawlers follow the redirect, and Google systems use the redirect as a weak signal that the redirect target should be processed. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 307 means the resource is at another address for now, and the request method stays the same.
The 307 (Temporary Redirect) status code indicates that the target resource resides temporarily under a different URI and the user agent MUST NOT change the request method if it performs an automatic redirection to that URI. RFC 9110: HTTP Semantics
Google says it treats a 307 as it treats a 302.
Equivalent to 302. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 303 sends the client to a different resource, which answers the request indirectly.
The 303 (See Other) status code indicates that the server is redirecting the user agent to a different resource, as indicated by a URI in the Location header field, which is intended to provide an indirect response to the original request. RFC 9110: HTTP Semantics
Google says its crawlers follow a 303 by default and use it as a weak signal to process the target.
By default, Google's crawlers follow the redirect, and Google systems use the redirect as a weak signal that the redirect target should be processed. Google: How HTTP status codes affect Google's crawlers
Google says it ignores the content of a redirecting URL and processes the final target.
Any content Google receives from the redirecting URL is ignored, and the final target URL's content is processed instead. Google: How HTTP status codes affect Google's crawlers
Google says it does not use the content of a URL that returns a 4xx status.
Google doesn't use the content from URLs that return 4xx status codes. Google: How HTTP status codes affect Google's crawlers
Google says Search does not index a URL that returns 4xx, and removes one it had indexed.
In the case of Google Search, Google doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index. Google: How HTTP status codes affect Google's crawlers
RFC 6585 says a 429 means too many requests arrived in a given time.
The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting"). RFC 6585: Additional HTTP Status Codes
Google says its crawlers read a 429 as an overloaded server and count it as a server error.
Google's crawlers treat the 429 status code as a signal that the server is overloaded, and it's considered a server error. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says a 503 means the server cannot handle the request for now.
The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay. RFC 9110: HTTP Semantics
Google says 5xx and 429 answers make its crawlers slow down for a while.
5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. Google: How HTTP status codes affect Google's crawlers
Google says Search keeps already indexed URLs that return 5xx at first and drops them in the end.
For Google Search, already indexed URLs are preserved in the index, but eventually dropped. Google: How HTTP status codes affect Google's crawlers

Location and the redirect chain

Google says its crawlers follow up to 10 redirect hops by default.
By default, Google's crawlers follow up to 10 redirect hops. Google: How HTTP status codes affect Google's crawlers
RFC 9110 says the Location value of a 3xx response is the address to redirect to.
For 3xx (Redirection) responses, the Location value refers to the preferred target resource for automatically redirecting the request. RFC 9110: HTTP Semantics
RFC 9110 says a relative Location value is resolved against the address that was requested.
When it has the form of a relative reference ([URI], Section 4.2), the final value is computed by resolving it against the target URI ([URI], Section 5). RFC 9110: HTTP Semantics

X-Robots-Tag

Google says any rule that works in a robots meta tag also works as an X-Robots-Tag header.
Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says the X-Robots-Tag header may name a user agent before the rules.
The X-Robots-Tag may optionally specify a user agent before the rules. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says rules with no user agent in front are valid for all crawlers.
Rules specified without a user agent are valid for all crawlers. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says that when rules conflict, the more restrictive one applies.
In the case of conflicting robots rules, the more restrictive rule applies. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says a crawler can read these settings only if it is allowed to fetch the page.
Keep in mind that these settings can be read and followed only if crawlers are allowed to access the pages that include these settings. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says other search engines may not treat these rules the same way.
It is possible that these rules may not be treated the same by all other search engines. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says all sets no restriction and is the default.
There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says noindex keeps the page, media or resource out of search results.
Do not show this page, media, or resource in search results. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says nofollow tells it not to follow the links on the page.
Do not follow the links on this page. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says none is the same as noindex with nofollow.
Equivalent to noindex, nofollow. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says nosnippet stops a text snippet or video preview for the page in search results.
Do not show a text snippet or video preview in the search results for this page. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says indexifembedded only has an effect next to noindex.
indexifembedded only has an effect if it's accompanied by noindex. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-snippet caps the text snippet at that many characters.
Use a maximum of [number] characters as a textual snippet for this search result. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-image-preview sets the largest image preview for the page in search results.
Set the maximum size of an image preview for this page in search results. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-video-preview caps a video snippet at that many seconds.
Use a maximum of [number] seconds as a video snippet for videos on this page in search results. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-snippet: 0 means no snippet is shown.
No snippet is to be shown. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says that with max-snippet: -1 it chooses the snippet length itself.
Google will choose the snippet length that it believes is most effective to help users discover your content and direct users to your site. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-image-preview: none means no image preview is shown.
No image preview is to be shown. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-image-preview: standard means a default image preview may be shown.
A default image preview may be shown. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-image-preview: large means a larger image preview may be shown.
A larger image preview, up to the width of the viewport, may be shown. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says that with max-video-preview: 0 at most a static image may be used.
At most, a static image may be used, in accordance to the max-image-preview setting. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says max-video-preview: -1 sets no limit.
There is no limit. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says notranslate stops it offering a translation of the page in search results.
Don't offer translation of this page in search results. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says noimageindex tells it not to index the images on the page.
Do not index images on this page. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says unavailable_after keeps the page out of search results after that date and time.
Do not show this page in search results after the specified date/time. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says Search no longer uses noarchive to control a cached link.
The noarchive rule is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says Search does not use nocache.
The nocache rule isn't used by Google Search. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google says Search no longer uses nositelinkssearchbox to control the sitelink search box.
The nositelinkssearchbox rule is no longer used by Google Search to control whether the sitelink search box is shown for a given page, as the feature no longer exists. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
RFC 8288 says the Link header carries one or more links in the HTTP headers.
The Link header field provides a means for serialising one or more links into HTTP headers. RFC 8288: Web Linking
RFC 6596 says canonical names the preferred address among resources with duplicate content.
The canonical link relation specifies the preferred IRI from resources with duplicative content. RFC 6596: The Canonical Link Relation
Google says it supports a canonical in the header for web search results only.
Google supports this method for web search results only. Google: How to specify a canonical URL with rel="canonical" and other methods
Google says to use an absolute URL in the canonical header.
As with the rel="canonical" link element, use absolute URLs in the rel="canonical" HTTP header. Google: How to specify a canonical URL with rel="canonical" and other methods

Content-Type

RFC 9110 says Content-Type gives the media type of what the response holds.
The "Content-Type" header field indicates the media type of the associated representation: either the representation enclosed in the message content or the selected representation, as determined by the message semantics. RFC 9110: HTTP Semantics
Google says to block indexing of a file that is not HTML with the X-Robots-Tag header.
To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead. Google: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications

Retry-After

RFC 9110 says Retry-After tells the client how long to wait before it asks again.
Servers send the "Retry-After" header field to indicate how long the user agent ought to wait before making a follow-up request. RFC 9110: HTTP Semantics
RFC 9110 says the Retry-After value is a date or a number of seconds.
The Retry-After field value can be either an HTTP-date or a number of seconds to delay after receiving the response. RFC 9110: HTTP Semantics
RFC 9110 says Retry-After with a 503 is how long the service expects to be unavailable.
When sent with a 503 (Service Unavailable) response, Retry-After indicates how long the service is expected to be unavailable to the client. RFC 9110: HTTP Semantics
RFC 6585 says a 429 may carry Retry-After to say how long to wait before a new request.
The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request. RFC 6585: Additional HTTP Status Codes
RFC 9110 says Retry-After with a redirect is the least time to wait before the redirected request.
When sent with any 3xx (Redirection) response, Retry-After indicates the minimum time that the user agent is asked to wait before issuing the redirected request. RFC 9110: HTTP Semantics
Google's page on status codes does not mention Retry-After.
We searched the page for "Retry-After" on 14 Nov 2026 and found no match. Google: How HTTP status codes affect Google's crawlers

cf-mitigated

Cloudflare says a challenge page sent in place of the expected response carries cf-mitigated, set to challenge.
When a request encounters a Cloudflare Challenge Page instead of the originally anticipated response, the Challenge Page response (regardless of the Challenge Page type) will have the cf-mitigated header present and set to challenge. Cloudflare: Detect a Challenge Page response
Cloudflare says a challenge page is always sent as text/html, whatever was requested.
Regardless of the requested resource-type, the content-type of a challenge will be text/html. Cloudflare: Detect a Challenge Page response
Cloudflare says challenge is the only valid value of cf-mitigated.
For the cf-mitigated header, challenge is the only valid value. Cloudflare: Detect a Challenge Page response

X-Robots-Tag and the AI crawler names

The sentences about X-Robots-Tag above are Google's, from its page for Google Search. Google says the X-Robots-Tag header may name a user agent before the rules. Google says rules with no user agent in front are valid for all crawlers.

When a response has an X-Robots-Tag line, the reader lists the rules it addresses to each name in our AI crawler tokens table: ClaudeBot, Claude-SearchBot, Claude-User, GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, Google-Extended, PerplexityBot, Perplexity-User. It shows what is written in the header. It does not say that any of those crawlers obeys it.

The names come from the crawler pages of Anthropic, OpenAI, Google and Perplexity. On 14 Nov 2026 we searched each of those pages for "X-Robots-Tag", "noindex", "Retry-After" and "cf-mitigated" and found no match. So the reader says of each name only that its vendor's page does not mention the header.

What it cannot tell you

It cannot tell you what a crawler does. It reads the lines you paste and repeats what a standard or a vendor's page says about them. A crawler may act another way.

It does not fetch anything. A server can answer one client differently from another, so the headers you paste are the ones your request got. It cannot see the page itself, so it misses a robots meta tag or a canonical link in the HTML. The instant audit fetches one page from our server and reads its HTML.

It does not read robots.txt. The robots.txt sandbox tests a path against the rules of a pasted file, for each AI crawler name.

It does not know the address you asked for. It lists each Location value as written and does not check that the next response came from it. To check a whole redirect map from a terminal, use one-hop.

What happens to the text you paste

It stays in your browser. The reader makes no request with it, and closing or reloading the tab discards it. The session recording and the click analytics on the rest of this site read text off a page, so this page does not load them.

There is no link to share a reading. Response headers often hold a cookie or a token, and a link would carry it.