[Cosmos] DiskANN vector query returns nondeterministic top-K on unchanged container (product quantizer; spherical unaffected) — likely service-side, filing for routing

[ASeabrook36]

Note on routing: I believe this is service-side (query engine / DiskANN traversal) rather than SDK behavior — the SDK faithfully returns what the service sends, and plan-level telemetry is identical across differing executions. Filing here as the nearest active public channel for the Python SDK; happy to be redirected, and happy to share a subscription ID privately for backend telemetry correlation.

Summary

The same vector query, issued repeatedly against the same unchanged container in immediate succession, sometimes returns a high-similarity document in its top-10 and sometimes does not. This occurs with the product quantizer at quantizationByteSize=256. Three containers built identically from identical data all show it. Three containers differing only by using the spherical quantizer show none of it across the same test.

Environment

• Azure Cosmos DB for NoSQL, serverless account, East US 2
• Container API version 2024-05-15; Python SDK azure-cosmos, AAD auth (AzureCliCredential)
• Six validation containers: three product-quantizer, three spherical, otherwise identical

Configuration (product arm)

DiskANN index; quantizerType: product; quantizationByteSize: 256; indexingSearchListSize: 100; vectorIndexShardKey on a string property; embedding dataType: float32, 3072 dimensions, cosine. Spherical arm identical except quantizerType: spherical (no byte-size property, per documented mutual exclusivity). Hierarchical partition key (MultiHash, 3 levels).

Data

112,000 synthetic documents per container; each embedding is a random unit vector in 3072 dims. 30 documents overwritten with vectors constructed at cosine similarity 0.92–0.98 to corresponding query vectors — one known high-similarity answer per test query over a near-orthogonal noise floor (~0.07). All six containers loaded from a byte-identical SHA-256-verified artifact, single-threaded, same order. Every load: 112,000/112,000 written, zero failures, zero 429s, identical total RU to the decimal.

Query

SELECT TOP 10 c.chunkId, c.representationClass,
       VectorDistance(c.embedding, @q, false, {'dataType':'float32'}) AS score
FROM c
WHERE c.tenantId = @tenant AND c.representationClass = @class
ORDER BY VectorDistance(c.embedding, @q, false, {'dataType':'float32'})

Identical query text and parameter values for every execution below.

Method

For each of the 30 seeded queries: the identical query issued 5 times back-to-back, single-threaded, one session (~7 minutes across all six containers). Index build confirmed complete first — x-ms-documentdb-collection-index-transformation-progress read 100 on all six before any query.

Results

Container	Quantizer	Queries identical across all 5 repeats	Queries where the answer appeared in only some repeats
Build A	product/256	2/30	2 (answer present 2/5 and 3/5)
Build B	product/256	5/30	4 (4/5, 3/5, 2/5, 3/5)
Build C	product/256	1/30	4 (2/5, 2/5, 3/5, 3/5)
Build A	spherical	30/30	0
Build B	spherical	8/30	0
Build C	spherical	30/30	0

Most set variation everywhere is reordering among documents scoring 0.04–0.08 (expected — near-orthogonal random vectors). The concerning cases are the last column: a document scoring ~0.95 against the query absent from the top-10 on some executions and present on others, same query, same container, minutes apart.

Supporting observations

1. An unscoped exhaustive scan (no shard-key predicate; retrievedDocumentCount = 112,000) returned all 30 planted answers on every container on every run — the documents are present, correctly stored, and in the true exact top-10, including on executions where the indexed query missed them.
2. On differing executions of the same query, retrievedDocumentCount and indexUtilizationRatio (0.02) were identical. The plan appears stable; the traversal result varies.

Questions

1. Is per-execution variation in DiskANN results expected for a static container — and is this magnitude (a ~0.95 document dropping from top-10 on ~1/3 of executions) within expected bounds?
2. Does anything in the environment plausibly explain it — background index maintenance on serverless, partition-level behavior, or an interaction between vectorIndexShardKey and product quantization at 256 bytes?
3. Is the product/spherical difference here consistent with their documented characteristics?

Available on request

Full per-execution logs (returned document IDs + scores), a ~200-line standalone reproduction script, and the preserved unmodified containers. Planning to write this up publicly; would rather do so with an accurate account of the cause — any correction welcome.

[github-actions]

Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.

//cc: @AbhinavTrips @bambriz @Pilchie @pjohari-ms @simorenoh

[github-actions]

🎯 Agentic Issue Triage

Summary: The author observes nondeterministic top-K results from repeated identical DiskANN vector queries on an unchanged Azure Cosmos DB for NoSQL container when using the product quantizer (quantizationByteSize=256), while the spherical quantizer shows consistent results. The author correctly notes this appears to be a service-side issue and is filing for routing.

📋 Issue Details

Package: azure-cosmos (AAD auth via AzureCliCredential), container API version 2024-05-15
Affected API: VectorDistance / DiskANN indexed vector queries with product quantizer
Scenarios:

• Same SELECT TOP 10 ... ORDER BY VectorDistance(...) query issued 5× back-to-back on static container
• A known high-similarity document (~0.95 cosine) drops from top-10 on ~1/3 of executions with product quantizer
• Spherical quantizer shows no such dropping across identical containers
• Exhaustive unscoped scan always returns all planted answers — documents are present and correctly stored

Root ask: Determine whether per-execution DiskANN variation of this magnitude is expected, and clarify whether the product/spherical difference, vectorIndexShardKey interaction, or serverless account behavior explains the observation.

🔎 Debugging / Reproduction Notes

The author has already done thorough isolation:

• Six identical containers (3 product, 3 spherical), byte-identical data loads, SHA-256 verified
• Exhaustive scan confirming all documents are present (rules out data load issues)
• retrievedDocumentCount and indexUtilizationRatio identical across diverging executions (stable plan)
• Index transformation confirmed at 100% before queries

Suggested investigation steps:

1. Review backend DiskANN traversal behavior for product quantizer at quantizationByteSize=256 on serverless accounts — is non-determinism an expected ANN property at this configuration?
2. Check whether vectorIndexShardKey combined with product quantization introduces partition-routing variance
3. Correlate with subscription/container telemetry (author is willing to share subscription ID privately)
4. Evaluate whether the ~0.95 document's miss rate (~1/3 executions) falls within or outside expected ANN recall bounds for this index configuration

🏷️ Label Confidence

• Category: Client — issue involves the azure-cosmos Python SDK and its interaction with the Cosmos DB service; data-plane client library
• Service: Cosmos — explicitly references Azure Cosmos DB for NoSQL and azure-cosmos SDK
• Confidence: High — package, service, and category are unambiguous; confirmed by reference issue Type hint is wrong for python azure-cosmos query parameters #40458 (Cosmos + Client pattern)

👥 Owner Routing

• Matched CODEOWNERS entry: # ServiceLabel: %Cosmos %Service Attention (line 522) — bottom-to-top scan; first match for "Cosmos" service label (ignoring %Service Attention token)
• AzureSdkOwners: none listed
• ServiceOwners: @AbhinavTrips @bambriz @pilchie @pjohari-ms @simorenoh
• Routing action: Added Service Attention and needs-team-attention labels; ServiceOwners mentioned via routing comment
• Scan notes: Line 194 (# ServiceLabel: %Cosmos) was also encountered but line 522 was reached first in bottom-to-top order and matched

Generated by Agentic Triage for issue #48414 · 29 AIC · ⌖ 6.89 AIC · ⊞ 9.6K · ◷

[harsha-simhadri]

ASeabrook36 Thanks for the detailed report.

• Vector indices (DiskANN included) are for approximate search, so not all good documents are guaranteed to be returned (ie. recall is not 100%). In practice, we aim to return, say 95% (depends on the query params), of good documents across queries.
• quantizedFlat is an exhaustive scan of all quantized vectors, so its far more likely to get close to 100% recall.
• Vector indices are in general good at indexing realistic embeddings from AI models, but poor at indexing randomly generated vectors. This is a property of all vector indexing algorithms. See, for example, https://arxiv.org/abs/1907.07387 for a more detailed explanation.
• In our backend, each replica builds an index independently, so might provide a different approximation. Queries can be served by any replica so some non-determinism is not unexpected. This is further amplified by the random vectors.

Suggested mitigations or revisions to experiments

• "random unit vector in 3072 dims."-> please generate vectors using a good embedding model. For 3072 dimensions, openai-text-3-large is a reasonable candidate.
• Use spherical quantizer which is better suited to openai-text-3-small/large embeddings. PQ should be OK too.
• If non-determism persists, increase search list multiplier to improve the depth of querying in the index.

In our benchmarks, the default parameters for the diskann+spherical quantization provide 98+% recall@10 for openai-text-3-small.

If your problem persists after these mitigations or you have a need for custom embeddings and/or determinism, please reach out harshasi@microsoft.com, and I am happen to discuss on a call.

[ASeabrook36]

Thank you, this is exactly the kind of answer I was hoping for, and the replica explanation checks out against my data.

Point fully taken on the corpus, it was an artificial setup for the purposes of the benchmark and I just thought the results were interesting enough to share/get feedback on. No real problem here. Per your suggestion I'll rerun with real model embeddings when I extend my prototyping a bit further.

One thought, if the replica-independence behavior isn't currently covered in the public vector search docs, it might be worth a line there.

[simorenoh]

ASeabrook36 we have updated our public documentation to call out the behavior you experienced, thanks!

https://learn.microsoft.com/azure/cosmos-db/vector-search#approximate-search-and-result-determinism