X-Robots-Tag Noindex (Header-Based) - Test #5

Test Page #5: X-Robots-Tag: noindex (HTTP Header)

This page has no meta robots tag in the HTML. Instead, it's blocked from indexing via the X-Robots-Tag HTTP response header.

Expected Issue: "Noindex + Internal Links" (detected via x_robots_tag field)

Test Configuration

HTTP Response Header (set via .htaccess):

X-Robots-Tag: noindex

.htaccess Configuration:

<Files "x-robots-noindex.html">
    Header set X-Robots-Tag "noindex"
</Files>

What This Tests:

Detection of X-Robots-Tag HTTP response header
Server-side noindex (vs. meta tag in HTML)
Proper handling of both noindex methods
Crawler must check HTTP headers, not just HTML

When X-Robots-Tag Is Used:

Non-HTML files (PDFs, images, etc.)
Bulk noindex via server configuration
When you can't modify HTML (third-party content)
Dynamic content where meta tags are complex

SEO Impact:

Same as meta noindex: This page won't be indexed by search engines. Any internal links pointing here are wasting link equity.

How to Verify:

Open browser DevTools (F12)
Go to Network tab
Reload this page
Click on the HTML document request
Look at Response Headers for X-Robots-Tag: noindex

Expected Detection

Issue Type:	Noindex + Links
Severity:	High
Noindex Source:	X-Robots-Tag header
Meta Robots:	Not set

Compare Methods

	Meta Tag	X-Robots
Location	HTML	HTTP Header
Non-HTML	No	Yes
Server Config	No	Yes

Navigation

Back to Test Hub Main Site

Page Source vs HTTP Headers

HTML Source (no noindex):

<head>
    <title>X-Robots-Tag Test</title>
    <!-- NO meta robots tag here! -->
</head>

HTTP Response Headers:

HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex
...