You know the feeling: you publish a new page, check Google Search Console a day later, and—nothing. The page hasn't even been crawled. Submitting a sitemap won't guarantee instant indexing, but it definitely tilts the odds in your favor. That's the real talk from Jes Scholz at Search Engine Journal, and it's exactly why sitemap optimization matters.
What Is an XML Sitemap?
An XML sitemap is a file that lists all of your website's URLs. It acts as a roadmap to tell the crawlers of indexing platforms—search engines, and yes, even large language models—what content is available and how to reach it. In the example above, a search engine will find all nine pages in a sitemap with one visit to the XML sitemap file. On the website, it would have to jump through five internal links on five pages to find page 9. This ability of XML sitemaps to assist crawlers in faster indexing is especially important for websites that have thousands of pages and/or a deep website architecture, frequently add new pages, frequently change the content of existing pages, suffer from weak internal linking and orphan pages, or lack a strong external link profile. Even though indexing platforms could technically find your URLs without it, by including pages in an XML sitemap, you're indicating that you consider them to be quality landing pages. And while there is no guarantee that an XML sitemap will get your pages crawled faster, let alone indexed or ranked, submitting one certainly increases your chances.
Creating a Sitemap: Static vs. Dynamic
There are two ways to create a sitemap: static sitemaps that must be manually updated, or dynamic sitemaps that are updated in real-time or by a regular cron job. Static sitemaps are simple to create using a tool such as Screaming Frog. The problem is that as soon as you create or remove a page, your sitemap is outdated. If you modify the content of a page, the sitemap won't automatically update the lastmod tag. So, unless you love manually creating and uploading sitemaps for every single change, it's best to avoid static sitemaps. Dynamic XML sitemaps, on the other hand, are automatically updated by your server to reflect relevant website changes. To create a dynamic XML sitemap you can ask your developer to code a custom script, use a dynamic sitemap generator tool, or install a plugin for your content management system, for example, the Yoast plugin for WordPress.
Building a Valid XML Sitemap
Your sitemap needs three items to introduce itself to indexing platforms. First, an XML version declaration that specifies the file type to inform indexing platforms what they can expect from the file. Second, UTF-8 encoding that ensures all the characters used can be understood. Third, specify the namespace that communicates what rules the sitemap follows. Most sitemaps use the "http://www.sitemaps.org/schemas/sitemap/0.9" namespace to show that the file conforms to standards set by sitemaps.org. Within each URL container, only two tags should be included for a URL. The loc tag, also known as the location tag, must contain the absolute, canonical version of the URL location. It should accurately reflect your site protocol (http or https) and whether you have chosen to include or exclude www. The lastmod tag, or last modified tag, is optional but highly recommended. It communicates the date and time the page was published or the last meaningful change. Google uses the lastmod value if it's consistently and verifiably accurate. Bing also uses lastmod to determine crawl frequency and index decisions. Mistakes, such as updating the lastmod value when the sitemap is generated rather than when the individual page was last modified, or worse, trying to manipulate crawlers by updating the date without significantly altering the page, may result in this signal being ignored by search engines. Do not include the changefreq or priority tags. Once upon a time, these hinted at how often to crawl, but search engines now ignore them entirely.
Sitemap Types: When to Use Which Kind
There are many different types of sitemaps, but you actually need only a few. An XML sitemap index becomes necessary when you exceed either the 50,000 URL limit or the 50 MB uncompressed size limit. Sitemaps can be compressed using gzip to save bandwidth, but once unzipped, the sitemap still can't exceed either limit. Whenever you exceed either limit, you will need to split your URLs across multiple XML sitemaps. Those sitemaps can then be combined into a single XML sitemap index file, often named sitemap-index.xml. Essentially, it is a sitemap for sitemaps. Submit your sitemap index to Google Search Console and Bing Webmaster Tools. Specify your sitemap or sitemap index URL(s) in your robots.txt file, pointing indexing platforms directly to your sitemap as you welcome them to crawl.
Image sitemaps were designed to improve the indexing of image content, but in modern SEO it's best practice to utilize JSON-LD schema.org/ImageObject markup instead, as it provides more attributes than an XML image sitemap. Because of this, a dedicated XML image sitemap is unnecessary. Simply add the image XML namespace declaration and the image tags directly to the main XML sitemap within the associated URL. Know that images don't have to be on the same domain as your website to be submitted in a sitemap; you can use a CDN as long as it's verified in Google Search Console.
Video sitemap tags can be added within an existing sitemap, but unlike images, video extensions offer a multitude of additional tags. If you leverage these tags extensively, consider a dedicated video sitemap. Adding these extensions increases the file size significantly and may lead to exceeding the file size limits. Either method will help Google discover, crawl, and index your video content as long as the video is related to the content of the host page and is accessible to Googlebot. While Bing does support video extensions in XML sitemaps, Fabrice Canel confirmed they prefer submission via IndexNow. Google News sitemaps can only be used for article content that was created in the last two days. Once the articles are older than 48 hours, remove the URLs from the sitemap. Contrary to some online advice, Google News sitemaps don't support image URLs. HTML sitemaps were designed to assist human users in finding content. If you have a good user experience and well-crafted internal links, you likely don't need an HTML sitemap. Check the page views of your HTML sitemap in Google Analytics—they're probably very low. If not, it's a good indication that you need to improve your website navigation. HTML sitemaps are generally linked in website footers, taking link equity from every single page of the website. Ask yourself if that's the best use of that link equity, or if you're including an HTML sitemap as a nod to legacy website best practices. If few humans use it, and indexing platforms don't need it as you have strong internal linking and an XML sitemap, does that HTML sitemap have a reason to exist? Many would argue no.
Optimizing Your Sitemap: Four Key Strategies
XML sitemap optimization involves how you structure your sitemaps and what URLs are included. How you choose to do this impacts how efficiently indexing platforms crawl your website and, thus, your content visibility. Here are four ways to optimize XML sitemaps.
First, only include SEO-relevant pages in XML sitemaps. An XML sitemap is a list of pages you want to be crawled—and subsequently given visibility by indexing platforms—which isn't necessarily every page of your website. A bot arrives at your website with an "allowance" for how many pages it will crawl. The XML sitemap indicates that you consider the included URLs more important than those that aren't blocked but not in the sitemap. You're using it to tell indexing platforms, "I'd really appreciate it if you'd focus on these URLs in particular." To help them crawl your site more intelligently and reap the benefits of faster (re)indexing, do not include 301 redirect URLs, 404 or 410 URLs, non-canonical URLs, pages with noindex tags, pages blocked by robots.txt, paginated pages, parameter URLs that aren't SEO-relevant, or resource pages accessible by a lead gen form (e.g., white paper PDFs). Utility pages that are useful to users but not intended to be landing pages—login page, contact us, privacy policy, account pages—should also be excluded. I've seen recommendations to add 3xx, 4xx, or non-indexable pages to sitemaps in the hope it will speed up deindexing, but similar to manipulation of the lastmod date, such attempts may result in the sitemaps being ignored by search engines, damaging your ability to have your valuable content efficiently crawled. Just because it's not in your XML sitemap doesn't necessarily mean that Google won't index those pages.
Second, ensure your XML sitemap is valid. XML sitemap validators can tell you if the XML code is valid, but this alone is not enough. There might be another reason why Google or Bing can't fetch your sitemap, such as robots directives. Third-party tools won't be able to identify this. The most efficient way to ensure your sitemap is valid is to submit it directly to Google Search Console and Bing Webmaster Tools. When valid in GSC and BWT, you will see the green "Success" status. If you get a red message instead, click on the error to find out why, fix it, and resubmit. But in Search Console and Bing Webmaster Tools, you can do so much more than simple validation.
Third, leverage sitemap reporting for indexing analysis. Say you submit 80,000 pages all in one sitemap index, and 9,000 are excluded by both Google and Bing. Sitemap reporting will help you understand the overarching why, but provides limited reporting on which URLs are problematic. So while it's valuable information, it's not easily actionable. You need to discover which types of pages were left out. What if you use descriptive sitemap names that reflect the sections of your website—one for categories, products, articles, etc.? Then, you can drill down to see that 7,000 of the 9,000 non-indexed URLs are category pages—and clearly know where to focus attention. This can also be done within a sitemap index file. Just remember, for this to effectively work, sitemaps need to be mutually exclusive, with each URL existing in only one sitemap. The exception is the Google News sitemap.
Fourth, strategize sitemap size. As mentioned before, search engines impose a limit of 50,000 URLs per sitemap file. Some SEO specialists intentionally reduce this number, say to 10,000, which can be helpful to speed up indexing. However, you can only download 1,000 URLs in GSC. So if 2,000 URLs in a certain sitemap are not indexed, you can only access half of them. If you are trying to do content cleanup, this will not be enough. To gain full visibility on all URLs causing issues, break sitemaps down into groups of 1,000. The obvious downside is that this has a higher setup time as all URLs need to be submitted in Google Search Console and Bing Webmaster Tools. This may also require high levels of ongoing management.
The Sitemap Best Practice Checklist
Do invest time to:
✓ Dynamically generate XML sitemaps. ✓ Compress sitemap files. ✓ Use a sitemap index file. ✓ Include the <loc> & <lastmod> tags. ✓ Use image tags in existing sitemaps. ✓ Use video and Google News sitemaps if relevant. ✓ Reference sitemap URLs in robots.txt. ✓ Submit sitemaps to both Google Search Console and Bing Webmaster Tools. ✓ Include only SEO-relevant pages in XML sitemaps. ✓ Ensure URLs are included only in a single sitemap. ✓ Ensure the sitemap code is error-free. ✓ Group URLs in descriptively named sitemaps based on page type. ✓ Strategize how to break down large sitemap files. ✓ Use Google Search Console and Bing Webmaster Tools to analyze indexing rates.
Now, go check your own sitemaps and make sure you're doing it right. The SEJ article makes clear: what was considered best practice yesterday does not hold true today, and this is especially relevant when it comes to XML sitemaps, which are almost as old as SEO itself. The problem is that it's time-consuming to sort valuable advice from all the misinformation on forums and social media about how to optimize XML sitemaps. But the finer details of implementing sitemaps to improve SEO performance may be missed if you don't dive in. What was considered best practice yesterday does not hold true today—and this is especially relevant when it comes to XML sitemaps, which are almost as old as SEO itself.
Source: Jes Scholz, Search Engine Journal, "How To Use XML Sitemaps To Boost SEO," February 19, 2025
