Vector index cache in Session (index_cache/metadata_cache) never invalidates on ObjectStore/credential change, causing indefinite use of stale/expired S3 credentials

[ZuoTiJia]

Summary

When a Dataset is opened with rotating/temporary credentials (e.g. S3 STS session tokens with an expiration), once a vector index has been opened and cached inside the process-wide Session.index_cache / Session.metadata_cache, all subsequent accesses to that index reuse the cached reader objects — which are bound to whichever ObjectStore (and thus whichever credentials) was active at cache-insertion time. This happens even after a fresh Dataset is built with new, valid credentials, because the cache lookup only keys on the index UUID and never revalidates against the object_store passed into the current call.

Since the cache backend (MokaCacheBackend, moka-based) has no time-based expiration at all (only capacity/weigher-based eviction), this stale binding can persist indefinitely — we observed it persisting for at least 12 hours with no self-healing. The only way to recover is to restart the process (which recreates the Session singleton).

Environment

• lance = "8.0.0" (crates.io)
• Used via a Rust FFI layer embedded in a host application (Apache Doris), with a lance::session::Session created once per process and reused via DatasetBuilder::from_uri(...).with_session(session) for every dataset open.
• Credentials are short-lived S3 session tokens issued per query and passed via storage_options (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN), refreshed on each new Dataset open.

Observed symptom

Intermittent KNN vector search failures:

Failed to calculate KNN: ... Generic S3 error: Error performing GET .../_indices/.../index.idx
... 400 Bad Request ... InvalidSessionToken

• Refreshing/reissuing credentials on the caller side and opening a brand-new Dataset does not fix it.
• Restarting the host process (which recreates the shared Session) immediately fixes it.
• Leaving the process idle for 12+ hours does not self-heal.

Root cause (verified by reading source)

1. Session.index_cache / Session.metadata_cache are backed by MokaCacheBackend:

   // lance-core/src/cache/moka.rs
   pub fn with_capacity(capacity: usize) -> Self {
       let cache = moka::future::Cache::builder()
           .max_capacity(capacity as u64)
           .weigher(|_, v: &MokaCacheEntry| v.size_bytes.try_into().unwrap_or(u32::MAX))
           .support_invalidation_closures()
           .build();
       Self { cache }
   }

   No .time_to_live() / .time_to_idle() is configured — entries only leave the cache via capacity-driven eviction or explicit clear().

2. In lance::index::vector::ivf::v2::reconstruct_typed(), the warm path for reconstructing a cached IvfIndexState looks up cached file readers purely by index UUID, ignoring the object_store passed in for this reconstruction:

   let readers_key = CachedIndexReadersKey { uuid: state.uuid.clone() };
   
   let (index_reader, aux_reader) =
       if let Some(cached) = file_metadata_cache.get_with_key(&readers_key).await {
           // Warm path: reuse the cached readers directly, no file opens needed.
           ((*cached.index_reader).clone(), (*cached.aux_reader).clone())
       } else {
           // Cold path: opens files using the *current* object_store
       };

   CachedIndexReadersKey has no dependency on which ObjectStore/credentials were used to build the cached readers. Once cached, the reader (and its embedded credentials) is reused forever regardless of what object_store is passed to later calls.

3. Similarly, in Dataset::open_generic_index() / open_vector_index() (lance/src/index.rs), a cache hit on IvfIndexStateCacheKey / LegacyVectorIndexCacheKey short-circuits and returns the previously cached index object directly, without ever reaching the code path that would rebuild it against the current Dataset's object_store:

   let state_key = IvfIndexStateCacheKey::new(uuid, frag_reuse_uuid.as_ref());
   if self.index_cache.get_with_key(&state_key).await.is_some() {
       let index = self.open_vector_index(column, uuid, metrics).await?;
       return Ok(index.as_index());
   }

4. Because Session is intended to be shared across multiple Dataset opens (per its own doc comment, to increase cache hit rate), and hosts naturally keep a single Session alive for the process lifetime, a Dataset reopened with fresh credentials still resolves to the same cached index/reader objects bound to the original (possibly now-expired) credentials.

This explains all observed symptoms: refreshing the caller's credentials/token doesn't help (the cache is inside Session, decoupled from any particular Dataset/token); restarting the process fixes it (new Session = empty caches); and it never self-heals over time (no TTL exists in this cache layer at all).

Suggested fix directions

• Include something that identifies the ObjectStore/credential generation (e.g. a hash of the effective storage options, or a credential "epoch") as part of the cache key for CachedIndexReadersKey, IvfIndexStateCacheKey, and LegacyVectorIndexCacheKey, so a credential change naturally produces a cache miss and forces a rebuild with the current object_store.
• Alternatively/additionally, expose a public API to invalidate Session.index_cache/metadata_cache entries associated with a specific dataset URI (there's already an internal session.index_cache.clear() used in tests, but nothing scoped to "this dataset's entries only" that's usable by downstream consumers reacting to a credential-related I/O error).
• Consider documenting explicitly that Session reuse across Dataset opens with rotating credentials is unsafe today, so users relying on temporary credentials are aware of this caching behavior.

Steps to reproduce

1. Create a `Session` and reuse it across multiple `DatasetBuilder::from_uri(...).with_session(session)` opens of the same dataset URI, each with different (but both currently valid at open-time) `storage_options` credentials — simulating credential rotation.
2. Run a vector/KNN query that opens and caches the index using the first credentials.
3. Invalidate the first credentials (e.g. expire the STS token) and open a *new* `Dataset` with the second (new, valid) credentials via the same `Session`.
4. Run the same KNN query again — observe that it still uses the first (now-expired) credentials internally and fails, despite the second `Dataset`/credentials being valid.

Expected behavior

No response

Lance version

8.0.0

Language binding

Rust

Environment

No response

Logs / traceback

[u70b3]

I would like to work on this. I plan to first add a regression test using one shared Session, the same dataset URI, and two ObjectStore instances representing credential rotation. Then I will evaluate an opaque ObjectStore generation identity in the vector index/reader cache namespace, without including credential secrets in cache keys.

[u70b3]

Implemented in #7944.

The fix assigns each live object store binding an opaque, clone-stable runtime identity and includes it only in store-bound vector cache keys (V2 open readers and legacy live indices). Serializable IVF state and materialized partitions remain reusable across credential generations.

Regression coverage exercises V3 and legacy indices with one shared session, the same URI, two static credential generations, two fragments, I/O generation assertions, recall >= 0.5, and credential-key leak checks.

[wjones127]

I'd like to propose an alternative solution: can we remove any credential references from a cache entry? That was my intention already with storing IvfIndexState: it's meant to just store the portable state and not any connection information.

[wjones127]

Coming back to this, if you have refreshing credentials, why aren't you configuring a credentials provider? It seems to me the root cause of the problem is you are handing Lance temporary credentials as if they were static credentials. The appropriate solution should be instead to pass a credentials provider or a storage options accessor:

https://docs.rs/lance-io/latest/lance_io/object_store/struct.ObjectStoreParams.html#method.with_aws_credentials

https://docs.rs/lance-io/latest/lance_io/object_store/struct.ObjectStoreParams.html#structfield.storage_options_accessor

That means the object store will automatically refresh and these cache entries will never become stale. Is there a reason that doesn't work for you?