Rebuild HNSW vector index when dimension or distance function changes#19046
Open
Akanksha-kedia wants to merge 1 commit into
Open
Rebuild HNSW vector index when dimension or distance function changes#19046Akanksha-kedia wants to merge 1 commit into
Akanksha-kedia wants to merge 1 commit into
Conversation
Write a lightweight metadata file (.vector.hnsw.metadata) alongside the HNSW Lucene index directory at seal() time, recording the vector dimension and Lucene VectorSimilarityFunction used to build the index. VectorIndexHandler.isVectorConfigChanged() reads this file and triggers a rebuild when dimension or distance function differs from the current table config. Legacy segments without a metadata file skip the check (backward-compatible) and get the file on their next rebuild. Changing dimension produces a structurally incompatible index; changing the distance function silently returns wrong query results. Neither failure is detectable without the stored config, so both require a deterministic rebuild rather than silent misuse of the wrong index. L2 and EUCLIDEAN are treated as equivalent (both map to Lucene's VectorSimilarityFunction.EUCLIDEAN) so no spurious rebuild occurs when switching between the two aliases. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
cc @Jackie-Jiang @richardstartin @npawar — would appreciate a review when you get a chance! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Extends
VectorIndexHandlerto detect and rebuild a HNSW vector index when the vector dimension or distance function changes in the table config — mirroring the same pattern used for bloom filter (#18898) and JSON index (#18920) rebuilds.Problem
Previously, if a table's vector index config changed (e.g. dimension or distance function), the existing index would be silently reused without rebuilding:
Neither failure was detectable without the stored build-time config.
Solution
Write a lightweight metadata file (
.vector.hnsw.metadata) alongside the HNSW Lucene index directory atseal()time, recording:vectorDimension(int)vectorSimilarityFunction(LuceneVectorSimilarityFunctionname)VectorIndexHandler.needUpdateIndices()reads this file and triggers a rebuild when either field differs from the current table config.Backward compatibility: Legacy segments without a metadata file skip the check and receive the file on their next rebuild — no forced migration.
Files changed
HnswVectorIndexCreator– writes.vector.hnsw.metadataat seal timeVectorIndexHandler– reads metadata file and triggers rebuild on mismatchVectorIndexUtils– helper for reading/writing the metadata fileV1Constants– addsVECTOR_HNSW_METADATA_FILE_EXTENSIONconstantVectorIndexHandlerTest– unit tests for all rebuild/skip branchesTest plan
VectorIndexHandlerTestcovers: dimension change → rebuild, distance-function change → rebuild, no change → skip, missing metadata → skip (backward compat), metadata present + matching → skip