X-Robots-Tag
The X-Robots-Tag is an HTTP response header that tells search engines how to crawl, index, or follow the links on a resource. Unlike the <meta name="robots"> tag—which only works in HTML documents—the X-Robots-Tag header can be applied to any file type (PDFs, images, videos, CSS, etc.), giving you fine-grained control over which resources search engines should include in their results.
Why use X-Robots-Tag
- File-type flexibility
Works on non-HTML files (PDFs, images, video files), whereas<meta>tags only apply to HTML pages. - Server-level control
Can be set or removed via server configuration (Apache, Nginx) or programmatically in application code. - Selective indexing
Control indexing on a per-directory or per-file basis without editing each document.
Common X-Robots-Tag directives
| Directive | Effect |
|---|---|
noindex | Prevents the resource from being indexed |
nofollow | Prevents search engines from following any links on the item |
noarchive | Disallows cached copy in search results |
nosnippet | Blocks text snippets or video previews in search results |
none | Equivalent to noindex, nofollow |
noimageindex | Prevents images on the resource from being indexed |
Example: Returning X-Robots-Tag via HTTP header
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, nofollow
X-Robots-Tag vs. Meta Robots Tag
| Feature | Meta Robots Tag | X-Robots-Tag Header |
|---|---|---|
| Applies to | HTML documents only | Any file type |
| Location | <head> section of HTML | HTTP response headers |
| Configuration method | Inline HTML editing | Server/app configuration |
| Use case | Page-level control | Global or file-type–level control |
In summary, the X-Robots-Tag header is a powerful tool for guiding search engine behavior across all types of resources, not just HTML pages. By adding directives like noindex or nofollow at the server level, you gain precise, centralized control over your site’s crawl and indexing rules.