Skip to content

Conversation

@roomote
Copy link
Contributor

@roomote roomote bot commented Nov 11, 2025

This PR attempts to address Issue #9169. Feedback and guidance are welcome.

Problem

When two configuration profiles are created with LiteLLM API provider but different instances (different base URLs), the list of models was always pulled from one of them, causing issues when they don't provide the same models.

Root Cause

The model cache was using only the provider name as the cache key, not distinguishing between different LiteLLM instances with different base URLs.

Solution

  • Added unique cache keys that include a hash of the base URL for providers that can have multiple instances (LiteLLM, Ollama, LM Studio, Requesty, DeepInfra)
  • Updated file cache naming to include instance-specific information
  • Improved cache flushing to handle all cache keys for a provider

Testing

  • All existing tests pass
  • The fix ensures that each LiteLLM instance maintains its own separate model cache

Fixes #9169


Important

Fixes model cache issue by using unique cache keys with base URL hash for LiteLLM and similar providers in modelCache.ts.

  • Behavior:
    • Fixes model cache issue for LiteLLM and similar providers by using unique cache keys with base URL hash in modelCache.ts.
    • Ensures each instance maintains its own model cache, preventing model mixing.
  • Functions:
    • Adds getCacheKey() and getCacheFilename() to generate unique cache keys and filenames.
    • Updates writeModels(), readModels(), and getModels() to use new cache key logic.
    • Modifies flushModels() to clear all cache keys for a provider.
  • Testing:
    • All existing tests pass, confirming separate model caches for each instance.

This description was created by Ellipsis for 2081cb1. You can customize this summary. It will automatically update as commits are pushed.

- Add unique cache keys that include base URL hash for providers with multiple instances
- Update file cache naming to include instance-specific information
- Improve cache flushing to handle all cache keys for a provider

Fixes #9169
@roomote roomote bot requested review from cte, jr and mrubens as code owners November 11, 2025 10:01
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. bug Something isn't working labels Nov 11, 2025
@roomote
Copy link
Contributor Author

roomote bot commented Nov 11, 2025

Rooviewer Clock   See task on Roo Cloud

Review completed. Found 2 issues that should be addressed:

  • Add "roo" provider to the cache key differentiation list (line 43) to prevent model mixing for multiple Roo instances
  • Update flushModels to also clear file cache entries, not just memory cache

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.


// For providers that can have multiple instances with different base URLs,
// include the base URL in the cache key
if (baseUrl && ["litellm", "ollama", "lmstudio", "requesty", "deepinfra"].includes(provider)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "roo" provider is missing from this list but also supports optional baseUrl (see line 166 in src/shared/api.ts). When users configure multiple Roo provider instances with different base URLs, their model caches will incorrectly share the same key, causing the same model mixing issue this PR aims to fix.

Fix it with Roo Code or mention @roomote and request a fix.

Comment on lines 178 to 187
export const flushModels = async (router: RouterName) => {
memoryCache.del(router)
// For providers that can have multiple instances, we need to flush all possible cache keys
// Since we don't know all the possible base URLs, we'll flush all keys that start with the provider name
const keys = memoryCache.keys()
for (const key of keys) {
if (key.startsWith(router)) {
memoryCache.del(key)
}
}
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flushModels function only clears the in-memory cache but doesn't remove corresponding file cache entries (created by writeModels at line 64). When a user flushes models for a provider with multiple instances, stale file caches will persist and could be read back by readModels (line 72), defeating the purpose of the flush operation. Consider iterating through cache files matching the provider pattern and deleting them, similar to how the memory cache keys are being cleared.

Fix it with Roo Code or mention @roomote and request a fix.

@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. size:M This PR changes 30-99 lines, ignoring generated files.

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

[BUG] Mixing models between configuration profiles

3 participants