The Noindex Tag Explained: When to Hide a Page From Google on Purpose
The noindex tag explained: what it does, how it differs from robots.txt disallow, and why pages sometimes get noindexed by accident.
DidYouSEO Team··7 min read
A site owner notices their internal search results pages, thank-you pages, and duplicate filter URLs are all showing up in Google, diluting their crawl budget and occasionally embarrassing them in search results. The fix isn't deleting the pages — it's one line telling Google not to index them at all. That line is the noindex tag, and most site owners have never deliberately set one, even though a well-run site usually needs a handful.
The noindex tag is one of the few SEO controls that's genuinely simple once you understand what it does and doesn't do — but it also causes real damage when it ends up on the wrong page by accident, which happens more often than most people realize.
What the noindex tag actually does
Noindex is a directive that tells search engines not to include a specific page in their search index. Google's own documentation on robots meta tags confirms that a noindexed page can still be crawled — Google's crawler visits it, reads it, follows its links — but the page itself will not appear in search results, no matter how many other sites link to it or how relevant it is.
That crawlable-but-not-indexable distinction matters. Noindex doesn't block access; it blocks appearance in results. A page can have thousands of visits from direct links or internal navigation and simply never surface in a Google search, which is exactly the point for pages that are useful to have live but useless to rank.
There are three ways to set it, and DidYouSEO's noindex checker tool checks all three because a page can technically have conflicting signals across them:
- A meta robots tag in the page's
<head>:<meta name="robots" content="noindex"> - A meta googlebot tag, which targets Google specifically and can differ from the general robots tag
- An X-Robots-Tag HTTP header, set server-side — the only option that works for non-HTML files like PDFs, images, or JavaScript, since you can't add a
<meta>tag inside those file types
Which pages actually deserve a noindex tag
Not every page needs to rank, and forcing every page into the index dilutes crawl priority on the pages that matter. Reasonable candidates:
- Internal search results pages —
/search?q=anythinggenerates infinite low-value URL variations that add nothing to search but multiply indefinitely - Thank-you / confirmation pages — useful for the user who just converted, meaningless as a search result
- Filtered or faceted navigation URLs — an ecommerce site with
?color=red&size=mediumcombinations can generate thousands of near-duplicate URLs from a handful of real products - Staging or internal-only pages that accidentally became publicly reachable
- Paginated duplicate content where the canonical tag alone isn't enough to prevent indexing of page 2, 3, 4 of a list
Moz's guide to noindex makes the useful distinction that noindex is for pages that should exist and be reachable, but shouldn't compete for rankings — as opposed to pages that shouldn't be crawled at all, which is a different tool.
Noindex vs. robots.txt disallow — they are not the same thing
This is the single most common confusion, and mixing them up actively backfires. robots.txt tells crawlers not to visit a page. Noindex tells crawlers "you can visit, but don't index." If you disallow a URL in robots.txt and try to noindex it, Google can't see the noindex tag at all — because it never crawled the page to read it. The page might stay indexed anyway if it has enough external links pointing to it, showing up in results with no title or description, which looks broken.
The correct pattern: if you want a page fully out of search results, use noindex and let it be crawled. Only use robots.txt disallow for pages you don't want crawled at all — regardless of indexing, like admin areas or infinite-parameter URLs eating crawl budget.
How pages get noindexed by accident
This is the more damaging failure mode, because it's invisible until traffic drops and someone investigates. Common causes:
A staging environment's noindex tag ships to production. Development and staging sites are often blanket-noindexed to keep unfinished work out of search. If that tag doesn't get stripped out during the production deploy, the live site — or a section of it — silently stops being indexable while looking completely normal to a human visitor.
A CMS or page builder plugin sets it by default. Some page builders and SEO plugins default new pages to noindex until manually reviewed, and that default gets missed on a page published in a hurry.
Conflicting signals across the three methods. A page might have noindex correctly removed from its meta tag but still carry a stale X-Robots-Tag header from a server config that wasn't updated — invisible in a normal page-source view, because headers don't show up when you view source in a browser.
One documented case is exactly this pattern: a team correctly noindexed a staging site during a rebuild, forgot to remove the tag before launch, and Google quietly stopped indexing the homepage for nearly a week — surfacing only when a competitor publicly noted they'd taken over the #1 ranking spot. It's the kind of mistake that stays completely silent until someone checks Search Console and sees "Excluded by 'noindex' tag" against a page that should be ranking.
How to check whether a page is noindexed
Since noindex can be set in three different places, and only one of them (the meta tag) is visible in a normal "view source," a proper check needs to look at all three at once:
| Signal | Where it lives | Visible in page source? |
|---|---|---|
| Meta robots tag | <head> of the HTML | Yes |
| Meta googlebot tag | <head> of the HTML | Yes |
| X-Robots-Tag header | HTTP response headers | No — requires a network/header inspection |
Run the URL through a free noindex checker to see all three at once, or check Google Search Console's URL Inspection tool, which shows exactly how Googlebot last saw the page's indexing status — including whether it's excluded and why.
Common mistakes with noindex
Noindexing a page and disallowing it in robots.txt at the same time. As covered above, this prevents Google from ever seeing the noindex directive, defeating the purpose entirely.
Forgetting to remove noindex after launch. A page built and tested with noindex on, then launched without flipping it off, is one of the most common silent traffic killers — especially on pages nobody thinks to double-check because they "look" published.
Using noindex instead of a canonical tag for near-duplicate content. If two pages are genuinely similar and one should consolidate into the other, a canonical tag is usually the better tool — it merges ranking signals onto one URL instead of just removing one from consideration.
Not checking after a redesign or platform migration. Noindex settings, like redirects, are easy to get right in a migration plan and wrong in the actual deployed config. Verify live, don't trust the spec document.
FAQ
What does the noindex tag do? It tells search engines not to include a specific page in search results, while still allowing the page to be crawled and its links followed. The page remains reachable by direct link or navigation; it simply won't surface in search.
Is noindex the same as robots.txt disallow? No. Robots.txt disallow prevents crawling entirely, so Google never sees the page's content or any noindex tag on it. Noindex allows crawling but blocks indexing. Using both together prevents Google from ever reading the noindex directive.
Can a noindexed page still get backlinks and traffic? Yes, from direct links, social shares, or internal navigation — noindex only affects whether the page appears in search engine results, not whether people can reach or link to it.
How do I know if my site has an accidental noindex tag? Check Google Search Console's Coverage report for pages marked "Excluded by 'noindex' tag," or run individual URLs through a noindex checker that inspects the meta tag, meta googlebot tag, and X-Robots-Tag header together.
Check your key pages aren't accidentally hidden
The pages that should rank and the pages that shouldn't are usually obvious in principle — the risk is a noindex tag surviving a deploy it wasn't meant to be part of. Run your most important URLs through the free noindex checker to confirm nothing important is quietly excluded from search.
See how your site scores - free
30+ SEO checks plus AI visibility. No credit card required.