X-Robots-Tag

The X-Robots-Tag is an HTTP RESPONSE HEADER that delivers crawling and indexing directives — like noindex (don't index this page), nofollow (don't follow its links), noarchive, and others — to search engine crawlers. It does the same job as the HTML META ROBOTS tag (the tag in a page's head), but because it's delivered in the HTTP header rather than the HTML, it works for ANY file type — including non-HTML files like PDFs, images, videos, and other documents that have no HTML head to put a meta tag in. So the X-Robots-Tag is the way to control indexing of non-HTML resources, and a flexible, server-level alternative to the meta robots tag for HTML too.

How it differs from the meta robots tag and when to use each: both the X-Robots-Tag and the meta robots tag deliver the same indexing/crawling DIRECTIVES (noindex to keep a page out of the index, nofollow to not pass link signals, noarchive to prevent caching, none, and combinations), and search engines honor them equivalently — the difference is the DELIVERY mechanism and what it enables. The meta robots tag lives in the HTML (so it only works for HTML pages — a PDF, image, or video file has no HTML head to hold it), while the X-Robots-Tag lives in the HTTP RESPONSE HEADER (delivered by the server with the file, so it works for ANY file type, HTML or not). This makes the X-Robots-Tag the tool for: controlling indexing of NON-HTML files (the key use — noindexing PDFs, spreadsheets, images, or other documents you don't want in search results, which the meta tag can't reach), applying directives at SCALE via server configuration (setting rules in the server config — Apache .htaccess, Nginx config — to apply noindex to whole directories, file types, or patterns at once, rather than editing each page's HTML), and as a server-level alternative for HTML pages (where editing the HTTP header is easier than editing the HTML — e.g., applying noindex across a section via server rules). The critical caveat (the crawl-vs-index distinction, shared with meta robots): for a noindex directive (in either form) to WORK, the crawler must be able to CRAWL the page and SEE the directive — so noindex must NOT be combined with a robots.txt DISALLOW that blocks crawling (if you block the URL in robots.txt, the crawler never fetches it and never sees the noindex header, so the page can still get indexed from external links without its content — the classic robots.txt-blocks-the-noindex mistake), and X-Robots-Tag noindex (like all noindex) is about INDEXING, not crawling (it keeps the page out of results but the crawler still fetches it to read the directive). The honest framing and connections: the X-Robots-Tag is the HTTP-header way to deliver indexing directives, valuable specifically because it works for any file type (the only way to noindex non-HTML resources) and at server-config scale, complementing the meta robots tag (for HTML) and distinct from robots.txt (which controls CRAWLING, not indexing — a different layer, and one that can break noindex if it blocks the crawl). The framing: the X-Robots-Tag delivers indexing and crawling directives (noindex, nofollow, etc.) in the HTTP response header — working for any file type including PDFs and images the meta robots tag can't reach, and applicable at server-config scale; the discipline is using it to control indexing of non-HTML resources and to apply directives efficiently via server rules, while respecting the crawl-vs-index distinction (never blocking a noindexed URL in robots.txt, or the crawler can't see the directive) and understanding it as the header-level complement to the meta robots tag, separate from robots.txt's crawl control.

The X-Robots-Tag matters most when you need to control indexing of NON-HTML files — noindexing PDFs, spreadsheets, images, videos, or other documents you don't want in search results, which the HTML meta robots tag cannot reach (no HTML head) — and when applying indexing directives at SCALE via server configuration (whole directories, file types, or patterns at once, rather than per-page HTML edits). It matters as the header-level tool in the technical-SEO indexing toolkit, complementing the meta robots tag (for HTML pages) and distinct from robots.txt (which controls crawling, not indexing). The discipline is using the X-Robots-Tag for non-HTML resources and server-config-scale directives, while respecting the crucial crawl-vs-index distinction — never blocking a noindexed URL in robots.txt (or the crawler can't fetch the file to see the noindex header, and it may get indexed anyway) — and understanding which layer each tool controls: X-Robots-Tag and meta robots for indexing, robots.txt for crawling.

The X-Robots-Tag was introduced by Google to deliver robots directives in the HTTP response header rather than HTML, extending indexing control to non-HTML files like PDFs and images that the meta robots tag can't reach and enabling server-config-scale rules; it became a standard technical-SEO tool, distinct from robots.txt's crawl control and subject to the same crawl-vs-index rule that a noindexed URL must stay crawlable.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where x-robots-tag is a core concern: