Noindex Checker
The Noindex Checker tests up to 10 URLs for a noindex directive and reports the exact value, its HTML or HTTP source, and the crawler it applies to. A noindex directive tells a search engine to exclude a URL from its index, so it decides index eligibility rather than crawl access.
What a noindex tag does
A noindex tag tells a search engine that a URL must not be included in its index, so the URL cannot appear in search results while the directive is present in the response. The tag and the X-Robots-Tag header carry the same instruction, and testing either one answers whether a URL is currently eligible for indexing.
Noindex is an indexing instruction rather than a crawling instruction. A crawler that receives noindex still fetched the URL, and the directive changes what happens to the content afterward.
A noindex directive reaches a search engine through one of two places: a robots meta element in the HTML, or an X-Robots-Tag field in the HTTP response header. The Noindex Checker reports which source supplied the value, because the correction differs for each.
Noindex markup in HTML and in HTTP headers
A noindex directive carries the same meaning in both sources, and the source decides how it is removed.
HTML meta robots element
<meta name="robots" content="noindex, nofollow"><meta name="googlebot" content="noindex">HTTP X-Robots-Tag response header
X-Robots-Tag: noindexX-Robots-Tag: googlebot: noindexNoindex tag and X-Robots-Tag header sources
A robots meta element applies to the HTML document that contains it, while an X-Robots-Tag response header applies to the response that carried it.
HTML meta robots directive
The HTML form declares the directive inside the document as a robots meta element, the element many tools and guides call the robots meta tag. It is read from the fetched HTML, so a template, SEO plugin, or page setting that emits the element controls the outcome.
X-Robots-Tag response header
The header form declares the directive on the response itself as an X-Robots-Tag field. It applies to non-HTML resources too, including PDFs and images, where no meta element can exist. Removing the HTML will not remove a header, and removing the header will not remove the HTML.
Crawler-specific directives
Both forms accept a crawler token such as googlebot, which narrows the directive to one crawler. A directive without a token applies to every crawler, so a Googlebot result and a general result answer different questions.
Noindex controls indexing, not crawling
Robots.txt controls whether a crawler may request a URL, and a noindex directive controls whether the fetched URL may enter the index. Blocking the request can prevent the fetch that would have delivered the directive.
Why a robots.txt block can hide a noindex directive
When robots.txt disallows a URL for a crawler, that crawler cannot read a page-level meta robots element or a response header, because it never receives the response. The blocked fetch and the missing directive are different facts.
Why a blocked page can stay in search results
A page can remain in search results after someone adds noindex behind a robots.txt block. A crawlable URL is the precondition for a noindex directive to take effect, which is why the two signals must be tested separately.
Which tool answers which question
Use the Robots.txt Tester for crawl access and the Noindex Checker for indexing directives. Testing one does not answer the other.
How to check whether a URL is noindexed
Fetch the live URL, read every robots meta element and every X-Robots-Tag field on that response, and record the directive value together with its source and applicable crawler.
Reading a result without a noindex directive
A result that contains no noindex directive means no supported directive was found in the inspected response. It does not establish that a search engine has indexed the URL, and it does not establish that a search engine will index it.
Reading a result with a noindex directive
The report names the source for a noindex result. An HTML source is corrected in the page or its template, and a header source is corrected in the application, proxy, or CDN configuration that added the field.
Confirming the correction
Retest the deployed response after the correction. A source-code change that has not been published leaves the fetched evidence unchanged. Once the directive is gone from the fetched response, confirm what the search engine has recorded with the Search Console URL Inspection tool, which shows the last crawl and whether the served page reported a noindex robots meta tag; requesting indexing after that refresh asks Google to recrawl the URL sooner than its own schedule.
When a noindex directive is intentional
Noindex suits URLs that exist for users but must not compete in search, including checkout and cart pages, internal search result pages, staging hosts, and low-value filtered or parameter URLs.
Common causes of an unintended directive
The directive becomes a problem when it reaches a URL that should rank. Sitewide noindex during a staging migration, a template default, and a plugin setting that was never reviewed are the recurring causes.
Conflicting index and noindex declarations
A conflicting pair such as index and noindex in one value, or an index meta element alongside a noindex header, is a configuration defect rather than a ranking decision. The restrictive directive governs the reported response until the contradiction is removed at its source.
Where a noindex directive can live
| Source | Where it lives | Applies to | How to remove it |
|---|---|---|---|
| Meta robots element | Fetched HTML document | That HTML page | Page setting, template, or SEO plugin that emits the element |
| X-Robots-Tag header | HTTP response headers | Any resource type, including PDFs and images | Application, proxy, or CDN configuration that adds the field |
| Crawler-token directive | Either source, with a crawler name | The named crawler only | The same source that declared the directive |
| Both sources at once | HTML element and header together | The reported response | Remove the declaration you do not intend at its own source |
Using the Noindex Checker
What the Noindex Checker checks and reports
Inspect every applicable robots meta element in the fetched HTML, including its crawler name and exact directive value, rather than inferring indexability from page content or a rendered preview.
Read every supported HTTP X-Robots-Tag field on the fetched response, including crawler-qualified values and non-HTML resources where a meta element cannot supply the directive.
Separate general robots instructions from crawler-specific directives, then show which values apply to the selected crawler so a Googlebot result is not presented as universal crawler behavior.
Keep blocked, unavailable and unsupported responses distinct from confirmed index or noindex evidence. An unsuccessful fetch remains inconclusive instead of being mislabeled as an indexable page.
When to use the Noindex Checker
Run the Noindex Checker when a page is unexpectedly absent from search or before a template change that affects indexing directives.
How to use the Noindex Checker
Example: inspect https://example.com/checkout to learn whether its fetched HTML or HTTP headers declare noindex for Googlebot.
Common Noindex Checker issues and fixes
When the evidence identifies a meta robots noindex value, remove it from the responsible page or shared template only if the URL should be eligible for indexing. Fetch the published page again to confirm the directive is gone rather than relying on a source-code change alone.
When X-Robots-Tag supplies the noindex instruction, correct the application, server, proxy or CDN configuration that added the header. Changing HTML will not remove HTTP evidence, so retest the deployed response and verify every applicable header value.
A robots.txt block and a noindex directive answer different questions. If Googlebot cannot crawl the URL, use the Robots.txt Tester to inspect the winning rule; do not treat this checker’s successful diagnostic fetch as proof that Googlebot can read the directive.
Understanding your Noindex Checker results
The report identifies applicable directives, their HTML or HTTP source, conflicts, robots.txt context and any indeterminate fetch state.
Limitation: The Noindex Checker reads raw fetched HTML and HTTP headers for up to 10 public URLs. It does not execute page JavaScript, reproduce every crawler’s cached response or prove whether a search engine has indexed the URL. Redirect, network, authentication, content-type and response-limit failures remain fetch evidence, not an SEO verdict. A result without noindex means only that no applicable supported directive was found in the inspected response; it does not guarantee crawling, canonical selection or indexing.
Frequently asked questions
Can robots.txt stop Google from seeing a noindex directive?
Yes. When robots.txt blocks Googlebot from fetching a URL, Googlebot cannot read a page-level meta robots directive or response header from that fetch. The report keeps robots.txt access separate from noindex evidence; test crawl access with the Robots.txt Tester before treating an unavailable directive as absent.
What happens when index and noindex both appear?
The Noindex Checker lists the applicable directives and flags the conflict. A restrictive noindex instruction takes precedence for the reported response, but the correction is to remove the contradictory declaration at its HTML or HTTP source and retest the fetched URL.
Does Google crawl a URL that carries noindex?
A crawler can still request a noindexed URL, because noindex changes what happens to the content after the fetch rather than whether the request is made. Google reads the directive from the response it fetched, so blocking that fetch in robots.txt prevents the directive from being read at all.
Does removing noindex restore indexing?
Removing the directive makes the URL eligible again, and eligibility is not the same as inclusion. A search engine still decides whether to index the page after it recrawls the corrected response, so confirm the published change and expect a delay between the fix and any new index state.
Does a result without noindex mean the URL is indexed?
No. The Noindex Checker reports what the inspected response declares about indexing. It does not read a search engine index, so an absent directive means only that no supported noindex evidence was found in that response.
Monitor URLs over time
Indexing directives change with template releases and plugin updates. Monitor your important URLs and get alerted when a directive changes.