The sitemap that quietly hid the whole blog
A hand-kept sitemap drifted, then a misread code comment silently dropped every blog post. The fix was to stop writing the sitemap by hand.
The problem
A hand-kept sitemap drifted, then silently dropped the entire blog.
The solution
Generate it from the built pages and blog data, and check every URL on deploy.
A sitemap is the list of pages you hand to Google. Most of the time nobody looks at it, which is how it can be wrong for months.
How it went wrong
The site I took over had a sitemap written by hand. It had 31 entries, used the bare domain instead of www, still listed old URL patterns, had one entry missing .com entirely and listed one page twice. Every time someone added a page, someone else had to remember to add it to the sitemap. Usually nobody did.
The worst failure was quieter. The site is built with Eleventy, and the sitemap logic skipped any page it believed was marked noindex. A code comment in the blog’s template was misread as that flag, so every blog post silently dropped out of the sitemap. Nothing errored, nothing turned red, and Google simply got a shorter list.
Later, the same kind of drift hit pagination: page 2 onwards of the blog listing was skipped.
The fix: derive it, don’t maintain it
The sitemap is now generated from two sources of truth that already have to be right for the site to work:
- The built pages. Every page that isn’t explicitly excluded (redirect stubs, thank-you pages, ads-only landing pages) is included.
- The blog taxonomy data. It feeds categories, topic hubs and paginated archives, so a new topic hub appears in the sitemap the moment it exists.
Three rules keep it honest:
- Exclusion is explicit. A page leaves the sitemap only through a real front-matter flag, never through text that happens to appear in a template.
- The canonical host is enforced. Every URL is built from one base address, so
wwwand the trailing slashes always match. - Every deploy checks what it shipped. A live check requests every URL in the new sitemap and fails loudly on anything that isn’t a 200.
The same idea now generates llms.txt, the plain-text index some AI tools read. It went from listing 66 URLs to all 107.
Results
- The sitemap grew from 38 URLs to 108 as the content programme shipped, with no manual edits.
- By late October, 106 of the 107 submitted pages were indexed.
- The blog, which was invisible in the sitemap for a while, now brings in more than half of all non-brand clicks.
Takeaway
If a file can be derived from data you already maintain, derive it. A hand-kept sitemap isn’t wrong on the day it’s written; it goes wrong slowly, and silently.