Guides · Discovery

XML Sitemap: What Belongs In It, and Why Yours Isn't Indexed

Last verified: October 2026 · About 9 minutes of reading

A sitemap is a list of URLs you want crawled, not a list of URLs that exist. Most sitemap problems come from that gap: the file is technically valid, and still asks Google to spend attention on pages it should never index.

What belongs in an XML sitemap

Only canonical URLs that return 200 and may be indexed, one host per file, with an honest last modification date.

That sentence is the filter: a URL that fails any of the three conditions does not belong in the file. A sitemap should contain the version of each page you want in search results, which means the URL that wins canonicalisation, not the parameterised variants that point at it. If a URL redirects, returns an error, or carries a noindex directive, it does not belong in the file.

The format also imposes a scope: all URLs in one sitemap must be on the same host, and unless the file is submitted through Search Console or declared in robots.txt, it only affects URLs below its own directory. That is why the conventional location is the site root: https://example.com/sitemap.xml covers everything.

A sitemap earns its keep in a narrow set of situations: a new site with few inbound links, a large catalogue where some pages sit many clicks from the homepage, or an inventory with orphan pages that nothing links to. On a small, well-linked site it changes little. That is not a reason to skip it, but it is a reason not to expect a ranking effect from adding one.

The hard limits

50,000 URLs or 50 MB uncompressed per file, and 50,000 sitemaps per index file.

The size limit is measured before compression, so gzipping a 60 MB file does not make it valid. The protocol requires entity escaping for every value, which in practice means & becomes & inside a query string. A single unescaped ampersand makes the XML invalid and the whole file unreadable, not just that URL.

A minimal correct file

Four required pieces: the XML declaration, the urlset namespace, one url element per page, and a loc inside each.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-25</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
  <url>
    <loc>https://example.com/search?q=plan&amp;page=2</loc>
  </url>
</urlset>

Note what is missing. There is no changefreq, no priority, and no trailing element to close the last URL. The namespace string still uses http:// even though the sitemap itself is served over HTTPS; that URI identifies the schema, it is not a link to fetch.

lastmod: what Google actually uses

lastmod should be the date the page's main content last changed, in W3C datetime form, and only if that stays accurate.

W3C datetime allows a date alone (2026-10-25) or a full timestamp with an offset (2026-10-25T09:30:00+00:00). Google's guidance adds a condition that matters more than the format: it uses the value when it is consistently and verifiably accurate, and stops trusting a site that stamps every URL with today's date. A footer wording change does not require a new lastmod; a rewritten section, new structured data or changed internal links does.

Google also states that it ignores priority and changefreq entirely. They remain valid in the protocol and harmless in the file, but writing them changes nothing. If your generator emits them, leave them alone or remove them; neither choice affects crawling.

What to leave out

Leave out redirects, errors, noindex pages, duplicates, session IDs and anything on another host.

• Redirecting URLs. List the destination, not the 301 that points at it. A sitemap full of redirects trains a crawler to distrust the file.
• 404 and 410 pages. Remove them when they disappear. A dead URL left in the file is a permanent error report.
• noindex pages. Listing a page you have told Google not to index is contradictory, and the fetch is wasted.
• Duplicate variants. One URL per piece of content. Tracking parameters, sort orders and print versions belong to the canonical, not to the sitemap.
• Other hosts. A sitemap at example.com cannot list blog.example.com; that host needs its own file.

The canonical question is separate from the sitemap question: the sitemap should list whichever URL the page declares, but the tag itself is covered in the canonical guide.

Submitting it: robots.txt, Search Console, and a retired endpoint

There are two working routes: a Sitemap line in robots.txt, and a one-time submission in Search Console.

Google deprecated the sitemap ping endpoint on 26 June 2023 and said it would stop functioning six months later; requests to it now fail. Any plugin, cron job or deploy script still firing that request is doing nothing, and the failure is silent unless something logs the response. Removing it is safe.

# robots.txt — announces the sitemap to every crawler that reads the file
Sitemap: https://example.com/sitemap.xml

The Sitemap line takes an absolute URL and can appear anywhere in the file, including between groups, because it must not terminate a group. Declaring it costs nothing and covers crawlers that never see your Search Console account. Search Console adds what robots.txt cannot: fetch status, parse errors and a per-sitemap indexed count. Submit the index file once if you use one; Google follows it to the children.

Sitemap index for large sites

An index lists sitemap files rather than pages, and it is the only supported way past 50,000 URLs.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemaps/blog.xml</loc>
    <lastmod>2026-10-25</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemaps/docs.xml</loc>
    <lastmod>2026-10-01</lastmod>
  </sitemap>
</sitemapindex>

Splitting by section is usually better than splitting by count. A blog sitemap and a docs sitemap can be regenerated on different schedules, and the Search Console report then tells you which section has the problem. Every child file must still obey the per-file limits, and every URL must still be on the same host as the index.

Why URLs stay "Discovered – currently not indexed"

The status means Google found the URL, judged it low priority, and chose not to crawl or index it yet.

This is the status that generates the most confusion, so a concrete example helps. This site's sitemap currently lists 25 URLs; 23 of them sit in "Discovered – currently not indexed". The file is valid, submitted, and fetched without errors — every technical check in this guide passes. The status persists because indexing is a quality and demand decision, not a file format decision: Google saw the URLs, and decided its crawl budget was better spent elsewhere.

That is also why the usual fixes do not apply. Resubmitting the sitemap changes nothing, because Google already has it. Removing and re-adding URLs changes nothing. What moves the status over time is the thing no sitemap controls: internal links from indexed pages, external links, and pages that answer a query someone actually types. A sitemap tells Google what exists; it cannot tell Google what matters.

How to verify a sitemap

Fetch the file, parse it as XML, and confirm the URLs inside return 200 without a noindex directive.

1. Fetch it. The response should be 200 with an XML content type, and the body should start with the XML declaration rather than an HTML error page.
2. Parse it. Any XML parser will reject unescaped ampersands and stray tags. Search Console does the same check and reports the line.
3. Sample the URLs. Pick ten entries and check status, canonical and robots meta. A sitemap that lists redirects or noindex pages is telling you the generator is wrong.
4. Compare counts. The number of URLs in the file should roughly match the number of indexable pages you believe you have. A large gap is a generator problem, not an indexing problem.

When Search Console says "Couldn't fetch"

"Couldn't fetch" means the file was never retrieved; it is a fetch problem, not a content problem.

The usual causes are a firewall or bot rule blocking Google's fetcher, a login or cookie redirect in front of the file, or a hostname mismatch — submitting www.example.com/sitemap.xml when the canonical host is example.com. Test by fetching the exact submitted URL yourself, following redirects, from outside your network. A status of "Success" with zero discovered URLs is a different problem: the file was retrieved and parsed, and the URLs inside it are the ones that need checking.

Frequently asked questions

Four questions cover indexing expectations, the retired ping, noindex pages and the size limit.

Last verified: October 2026. Limits and deprecations were checked against sitemaps.org and Google Search Central on 2026-10-25; other search engines may still support the ping endpoint, but Google does not.