The Robots Meta Tag

The robots meta tag is a line of HTML placed in the head of a single web page that gives search-engine crawlers instructions about that specific page. It takes the form of a meta tag whose name is robots and whose content is one or more directives. The two most common pairs are index or noindex, which say whether the page may appear in search results, and follow or nofollow, which say whether the crawler should follow the links on the page and pass authority through them. So a page marked noindex, follow will be kept out of the index while its links are still crawled; a page marked index, follow — the default when no tag is present — behaves normally. Other directives exist too, such as noarchive to prevent a cached copy and nosnippet to suppress the text preview.

The robots meta tag matters because it is the precise, page-level lever for controlling how a site appears in search. Plenty of pages should exist for users but not clutter the index — internal search-result pages, thank-you and confirmation pages, thin filter permutations, staging content, or printer-friendly duplicates. A noindex robots meta tag keeps those out of search results without hiding them from people. Getting this wrong is costly in both directions: a stray noindex on an important page quietly erases it from search, while forgetting to noindex low-value pages lets them dilute a site's quality signals. Because it lives in the page itself, the tag travels with the page and applies exactly where it is placed, which makes it both powerful and easy to misapply if it is copied onto templates without thought.

The robots meta tag is often confused with robots.txt, but they do different jobs. Robots.txt is a single file at the root of a site that tells crawlers which paths they may or may not crawl — it governs access, at scale, before a page is fetched. The robots meta tag lives inside an individual page and governs what happens after the page is crawled: whether it is indexed and whether its links are followed. The crucial subtlety is that blocking a page in robots.txt can prevent a crawler from ever seeing its robots meta tag, so a page disallowed in robots.txt can still appear in results (as a bare URL) precisely because the crawler was never allowed to read the noindex instruction inside it. To reliably keep a page out of the index, you let it be crawled and apply noindex, rather than blocking it in robots.txt.

The X-Robots-Tag is the third member of the family and the one that resolves a limitation of the meta tag. The robots meta tag only works in HTML documents, because it is an HTML element. The X-Robots-Tag delivers the same directives — noindex, nofollow, noarchive, and the rest — as an HTTP response header, which means it can be applied to non-HTML files such as PDFs, images, and other documents that have no head to put a meta tag in. It can also be set server-wide by pattern, making it efficient for applying rules across many files at once. In short: robots.txt controls crawling of paths, the robots meta tag controls indexing of HTML pages from inside the markup, and the X-Robots-Tag controls indexing of any response from the HTTP header. Choosing the right one for the job prevents both accidental exposure and accidental deindexing.

Using the robots meta tag well begins with a simple rule that trips up even experienced teams. If you want a page kept out of search, let crawlers reach it and apply noindex, rather than blocking it in robots.txt, because a blocked page can never have its noindex read. Decide deliberately which pages belong in the index and which do not. Thank-you pages, internal search results, thin filter permutations, and staging content usually deserve a noindex, while your real product and content pages should carry the default index, follow behavior. Be especially careful with templates, since a single noindex baked into a shared layout can silently remove a whole section of a site from search. Check the tag on a sample of live pages after any template change, because the failure is quiet and easy to miss.

The tag is powerful precisely because it is page-specific, so treat it as a per-page decision rather than a global switch. When a directive needs to apply to non-HTML files such as PDFs or images, reach for the X-Robots-Tag header instead, since the meta tag has no place to live in those responses. When you need to control crawling of whole paths at scale, that is robots.txt territory, not the meta tag. Match the tool to the job: robots.txt for crawl access to paths, the robots meta tag for indexing of individual HTML pages, and X-Robots-Tag for indexing of any response by header. Getting that mapping right is what prevents the two classic disasters — important pages vanishing under a stray noindex, and low-value pages cluttering the index because their noindex was never crawlable.

The robots meta tag is a standard HTML directive letting page authors control search-engine indexing and link-following at the page level.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where the robots meta tag is a core concern: