XML Sitemap Best Practices (and Common Mistakes That Hide Pages from Google)

A sitemap that Google trusts gets your pages crawled faster. A sitemap full of noise teaches Google to stop trusting your sitemap entirely — and that distrust doesn't stay contained to the bad URLs, it bleeds into how much attention the good ones get too.

Following XML sitemap best practices isn't about ticking a box during setup. It's an ongoing hygiene job, because sitemaps quietly rot as sites grow: pages get noindexed but never removed, CMS plugins hardcode a lastmod date that never actually reflects a change, and nobody notices until crawl stats in Search Console start looking strange.

Here's what actually belongs in a sitemap, and the four mistakes that undo it.

What actually belongs in a sitemap

A sitemap is a list of URLs you want a search engine to know about and prioritize crawling. It's a hint, not a guarantee of indexing — but Google's own guidance treats it as a meaningful signal for discovery, especially on large or poorly-linked sites, per Google's sitemap documentation.

Every URL you include should meet three conditions:

• Returns a 200 status. Not a redirect, not a 404.
• Is indexable. No noindex tag, no Disallow in robots.txt.
• Is the canonical version. Not a parameter variant, not a duplicate.

If a URL fails any of those three, it doesn't belong in the sitemap — full stop. Backlinko's sitemap guide puts it plainly: sitemaps are built for search engines, not humans, so there's no reason to pad one with pages you don't actually want ranked.

Mistake #1: noindex'd pages sitting in the sitemap

This is the single most common issue we see when running a technical SEO audit: pages tagged noindex still listed in sitemap.xml. It's a direct contradiction — the sitemap says "please index this," the meta tag says "don't." Google resolves the conflict by trusting the noindex tag, but it still spent a crawl on a URL that was never going anywhere, which is wasted budget on sites where budget is genuinely scarce.

This usually happens because sitemap generation and the noindex decision live in different parts of the CMS and nobody wired them together. If a page gets noindexed post-launch (thank-you pages, staging leftovers, filtered category pages), it needs to be pulled from the sitemap in the same step.

Mistake #2: stale or fabricated lastmod dates

The <lastmod> tag is supposed to tell Google when a page was last meaningfully changed, which helps Google prioritize recrawling fresher content. In practice, a lot of CMS plugins set it to today's date on every single crawl, whether or not anything actually changed.

Google has been direct about the consequence. John Mueller has said on record, multiple times going back to 2017 and reiterated as recently as 2024, that Google checks whether your lastmod values are trustworthy — and if a site has a history of inaccurate ones, Google stops trusting the tag site-wide. It's not graded page by page. Fake the date on a handful of pages to nudge a recrawl, and you can lose the signal's value across the entire domain.

Only update lastmod when you change something that actually matters — body content, structured data, primary links — not when a footer date auto-refreshes.

Mistake #3: exceeding the 50,000 URL / 50MB limit

The sitemap protocol caps every individual sitemap file at 50,000 URLs and 50MB uncompressed, according to the official sitemaps.org protocol specification. Go past either limit and the file is invalid — not "less effective," invalid.

Sites with more URLs than that need a sitemap index file: a parent file that lists multiple child sitemaps, each staying under the cap. Google's documentation on managing sitemaps at scale covers the index-file format directly. A practical pattern for larger sites: split by content type (blog, product, category) so that when something breaks, you know immediately which slice of the site is affected.

Gzip compression is allowed for bandwidth, but the limit applies to the file's uncompressed size — compressing a 60MB file to 40MB doesn't make it compliant.

Mistake #4: sitemap not referenced in robots.txt

A sitemap that exists but isn't declared anywhere is a sitemap Google has to stumble onto or wait for you to submit manually in Search Console. The fix is a single line:

Sitemap: https://example.com/sitemap.xml

Add it anywhere in robots.txt — you can list multiple sitemap lines if you have more than one, and there's no cap on how many you declare. Validate the whole file, including the sitemap reference, with our free robots.txt tester — it's a common spot for a typo'd path or an accidental trailing slash to break the reference silently.

A quick reference table

| Check | Rule | What happens if you miss it | |---|---|---| | noindex pages | Never include | Wasted crawl on pages that can't be indexed anyway | | lastmod accuracy | Only update on real content changes | Google stops trusting the tag site-wide | | File size | Under 50,000 URLs / 50MB uncompressed | File becomes invalid, silently ignored | | robots.txt reference | Sitemap: line present | Google has to discover the file on its own | | Canonical URLs only | No parameter or duplicate variants | Diluted signals, indexing confusion |

Fixing it in practice

Run your live sitemap through our free sitemap checker — it flags dead URLs, redirect chains, and noindex conflicts in one pass instead of you eyeballing an XML file by hand. For sites where the sitemap issue turns out to be one symptom of a bigger crawl problem, our guide to fixing "discovered — currently not indexed" walks through the rest of the crawl-budget picture.

FAQ

Do I need a sitemap if my site is small? Not strictly — Google can discover a small, well-linked site through crawling alone. But a sitemap costs almost nothing to maintain and removes any doubt about which URLs you consider canonical, so it's worth having even at a few dozen pages.

Does priority or changefreq in my sitemap actually matter? No. Google has stated it ignores both fields. They're part of the sitemap protocol and some generators still output them, but don't spend time tuning values Google doesn't read.

Can a bad sitemap hurt my rankings? Not directly as a penalty, but indirectly — yes. A sitemap full of untrustworthy signals wastes crawl attention and can delay indexing of pages that genuinely deserve it.

How often should I regenerate my sitemap? Automatically, on publish/edit/unpublish, rather than on a fixed schedule. A sitemap that lags your actual content by days is stale by definition.

Try it on your own site

Paste your sitemap URL into our free sitemap checker and see exactly which of these four mistakes, if any, are quietly hiding pages from Google right now.