DOC-6979 Publish the versioned pages in sitemap.xml

[paoloredis]

3,134 live docs pages appear in no sitemap: every page under a versioned directory (operate/rs, operate/kubernetes, develop/ai/redisvl). The published sitemap carries 2,682 URLs where a full build renders 5,816, and the gap is exactly the versioned set -- verified by diffing the deployed file against a local unrestricted build, which reconciles to the URL with no remainder.

Nothing was broken in the generator. The file fell between two jobs:

• The latest build rm -rfs the version directories before Hugo runs, so its sitemap cannot list a versioned page.
• Each versioned matrix build does render them, but its deploy uploads only the versioned subdirectory (output/operate/rs/${version}), and Hugo writes sitemap.xml at the root of output/ -- one level above anything that ships.

So both halves exist at build time and neither reaches the bucket.

Fix, in two parts:

• generate_version_sitemap.py runs in each versioned build and filters that build's sitemap down to its own subtree, writing the result into the directory that already deploys. No deploy command changes.
• merge_sitemaps.py runs in the new deploy_complete_sitemap job, unions the 39 versioned sitemaps with the latest build's, and overwrites the published sitemap.xml at both mirrors deploy_latest writes.

Taking URLs from Hugo's own output rather than deriving them from url: frontmatter is load-bearing. All 3,152 versioned .md files carry an explicit url:, but 18 are drafts; deriving would have published 18 URLs that 404. Copying whole elements also preserves the git-derived lastmod.

Two guards, both because a silent partial sitemap would recreate this bug:

• generate_version_sitemap.py exits non-zero when a subtree matches nothing, which is what a regression in the versioned url: scheme would look like.
• merge_sitemaps.py refuses to write unless it sees one sitemap per discovered version plus the latest build's, so a failed matrix build leaves the published file alone instead of trimming it.

Subtree matching is segment-anchored, not substring. operate/kubernetes/8.0 and operate/kubernetes/8.0.18 are both live version directories, so a substring match folds 78 pages of 8.0.18 into 8.0 and publishes them under the wrong version. test_sitemaps.py pins that case.

One flat urlset rather than a sitemap index: the SEO team's file is itself a sitemap index and the protocol forbids nesting one inside another. Overwriting the address they already reference also means no change on their side.

Note for whoever picks up the related SEO report: the 2,108 URLs it lists as "missing from the sitemap" are a different problem. Spot-checked, they are mostly anchor URLs and pre-restructure paths served by Hugo alias stubs, which return 200 with a meta-refresh -- which is why their liveness check passed. Redirects do not belong in a sitemap; this change does not address them, and the 200-instead-of-301 alias behaviour is worth its own ticket.

---

Note

Medium Risk
Changes production GCS publishing of sitemap.xml and adds a new deploy job, but scope is SEO artifacts only with explicit guards against partial merges.

Overview
Fixes a gap where ~3,100 versioned docs URLs never appeared in the published sitemap.xml, because the latest Hugo build strips version trees and versioned deploys only rsync a subdirectory (so root sitemap.xml never ships).

Each Kubernetes / RS / RDI / RedisVL matrix build now runs build/generate_version_sitemap.py after Hugo to write a per-version subtree sitemap into the directory that already deploys, and uploads that file as a lightweight artifact. A new deploy_complete_sitemap job merges the latest build’s sitemap with every version artifact via build/merge_sitemaps.py and overwrites sitemap.xml at both GCS mirrors SEO already uses (~5,800 URLs in one flat <urlset>).

Regression guards: empty subtree matches fail the version build; merge refuses to publish if the expected sitemap count is missing. build/test_sitemaps.py covers segment-anchored subtree matching (e.g. 8.0 vs 8.0.18) and merge/dedup behavior.

^{Reviewed by Cursor Bugbot for commit 47b73f2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

DOC-6979

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

🧠 Redis Memory

Found 5 related items from repository history (5 new this commit):

• Commit — DOC-6951 Publish redirects so a moved page is not read as a deletion 🆕

  DOC-6951 Publish redirects so a moved page is not read as a deletion

• PR — DOC-6951 Publish redirects so a moved page is not read as a deletion 🆕

  Makes a moved page resolvable instead of looking deleted — item D of DOC-6951. Based

• Commit — DOC-6954 Give the Client tools page its categories back 🆕

  DOC-6954 Give the Client tools page its categories back

• Commit — DOC-6954 Give the Client tools page its categories back 🆕

  DOC-6954 Give the Client tools page its categories back

• Commit — DOC-6951 Backfill 254 missing aliases for pages that moved 🆕

  DOC-6951 Backfill 254 missing aliases for pages that moved

Memory updated at 47b73f2

[kaitlynmichael]

Will this muddy up Google search results if all the versioned ones show up too?

[paoloredis]

@kaitlynmichael these changes were requested by the web team. Here's their explanation as to why these changes are needed:

A sitemap is the list a website hands Google saying "here is everything worth looking at." Google will still find pages without one by following links, but it finds them later and treats them with less confidence.