feat: add LiteLLM as unified LLM provider by RheagalFire · Pull Request #3182 · ModelEngine-Group/nexent

RheagalFire · 2026-06-02T18:23:07Z

Summary

Adds LiteLLM as a model provider, giving nexent users access to 100+ LLM providers (OpenAI, Anthropic, Google Gemini, Azure, Bedrock, Ollama, etc.) through litellm.completion() as an SDK dependency.
Two integration points: backend provider for model discovery + SDK model for inference.

Changes

backend/consts/provider.py: added LITELLM = "litellm" to ProviderEnum
backend/services/providers/litellm_provider.py: new LiteLLMModelProvider for model discovery via /v1/models
backend/services/model_provider_service.py: wired LiteLLM into get_provider_models() dispatch
sdk/nexent/core/models/litellm_llm.py: new LiteLLMModel with streaming, token tracking, tool calling, drop_params=True
sdk/nexent/core/models/__init__.py: registered LiteLLMModel
test/sdk/test_litellm_model.py: 17 tests (init, streaming, credentials, edge cases, live E2E)

Tests

Unit tests (17/17 pass):

TestInit::test_basic PASSED
TestInit::test_with_credentials PASSED
TestCall::test_streaming_output PASSED
TestCall::test_api_key_forwarded PASSED
TestCall::test_api_key_omitted_when_none PASSED
TestCall::test_api_base_forwarded PASSED
TestCall::test_drop_params_set PASSED
TestCall::test_response_format PASSED
TestCall::test_stop_sequences PASSED
TestCall::test_token_tracking PASSED
TestEdgeCases::test_empty_stream PASSED
TestEdgeCases::test_chunk_without_choices PASSED
TestEdgeCases::test_context_length_exceeded PASSED
TestEdgeCases::test_auth_error_propagates PASSED
TestEdgeCases::test_import_error PASSED
TestEdgeCases::test_stop_event_interrupts PASSED
TestLiveE2E::test_live_streaming PASSED
============================== 17 passed in 3.64s ==============================

Live E2E (Anthropic claude-sonnet-4-6 via Azure Foundry, streaming):

Live E2E response: 'OK'

Risk / Compatibility

Additive only. Existing providers and OpenAIModel untouched.
litellm is lazy-imported so the base install is unaffected.
drop_params=True ensures cross-provider compatibility.

Example usage

from nexent.core.models import LiteLLMModel

# Any LiteLLM-supported provider
model = LiteLLMModel(
    model_id="anthropic/claude-sonnet-4-20250514",
    api_key="sk-ant-...",
)

# Or Google Gemini
model = LiteLLMModel(model_id="gemini/gemini-2.5-flash")
# export GEMINI_API_KEY=...

response = model([{"role": "user", "content": "Hello!"}])
print(response.content)

RheagalFire · 2026-06-02T18:23:38Z

cc @Dallas98 @WMC001

JasonW404 · 2026-06-24T04:02:01Z

+                headers["Authorization"] = f"Bearer {api_key}"
+
+            models_url = f"{base_url}/models"
+


安全风险：verify=False 禁用了 SSL 证书验证，生产环境下容易遭受 MITM 攻击。项目已有 ssl_verify 配置字段，应该从 provider_config 读取并默认为 True。

JasonW404 · 2026-06-24T04:02:05Z

+
+            model_list = []
+            for item in data:
+                model_id = item.get("id", "")


所有通过 LiteLLM 发现的模型都使用同一个 DEFAULT_LLM_MAX_TOKENS，不区分实际能力。gpt-4o 有 128K context，gpt-3.5-turbo 只有 16K。这会与 PR #3293 的 capacity management 系统冲突。建议从 LiteLLM 的 model cost map 查询实际能力，或允许 operator 覆盖。

JasonW404 · 2026-06-24T04:02:08Z

+                "litellm is required for LiteLLMModel. "
+                "Install it with: pip install 'litellm>=1.80,<1.87'"
+            ) from e
+


ImportError 在 __call__ 时才抛出，意味着错误只在第一次调用时才暴露。建议在 __init__ 时就验证 litellm 是否可用，实现更早的失败反馈。

YehongPan · 2026-06-24T05:10:36Z

+
+            return model_list
+
+        except Exception as e:


[代码规范] except Exception: 过于宽泛，建议捕获更具体的异常类型，避免掩盖潜在错误。

YehongPan · 2026-06-24T05:10:38Z

+            message.role = MessageRole.ASSISTANT
+            return message
+
+        except Exception as e:


[代码规范] except Exception: 过于宽泛，建议捕获更具体的异常类型，避免掩盖潜在错误。

YehongPan · 2026-06-24T05:10:40Z

+
+            await litellm.acompletion(**kwargs)
+            return True
+        except Exception as e:


[代码规范] except Exception: 过于宽泛，建议捕获更具体的异常类型，避免掩盖潜在错误。

WMC001 · 2026-06-24T07:18:13Z

LGTM. LiteLLM integration follows the existing provider pattern. No issues found.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,77 @@
+import logging


SSL Verification Disabled

litellm_provider.py uses verify=False in httpx.AsyncClient. Exposes application to MITM attacks. Should use proper certificate validation or make configurable.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


OpenAI-Specific Stream Options

litellm_llm.py hardcodes stream_options:{include_usage:true} which is OpenAI-specific. May cause errors with non-OpenAI providers. Should conditionally apply based on provider.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


Token Counting Assumption

Token extraction assumes usage data is in the last chunk. Some providers may return usage in different positions. Should scan all chunks or use LiteLLM built-in tracking.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


Narrow Error Handling

Streaming error handling only catches context_length_exceeded. LiteLLM can throw various errors (rate limits, auth failures). Should catch broader exception types.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,77 @@
+import logging


Missing URL Validation

litellm_provider.py lacks URL validation for base_url. Invalid URLs cause cryptic httpx errors. Should validate URL format and provide clear error messages.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


Missing Dependency Declaration

LiteLLM not declared in sdk/pyproject.toml or backend/pyproject.toml. Import error suggests pip install but constraint not enforced in package metadata.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


Async Pattern Inconsistency

call is synchronous while check_connectivity is async. Should align with existing provider interface or document the deviation.

wuyuanfr · 2026-06-27T09:50:13Z

@@ -0,0 +1,201 @@
+"""LiteLLM-backed LLM model for nexent.


Resource Cleanup Missing

Stop event handling raises RuntimeError but doesn't close streaming iterator or clean up observer state. Should ensure cleanup in finally block.

feat: add LiteLLM as unified LLM provider

90979ac

RheagalFire requested review from Dallas98 and WMC001 as code owners June 2, 2026 18:23

test: add comprehensive tests for LiteLLMModel

776f886

JasonW404 reviewed Jun 24, 2026

View reviewed changes

YehongPan reviewed Jun 24, 2026

View reviewed changes

wuyuanfr reviewed Jun 27, 2026

View reviewed changes

		headers["Authorization"] = f"Bearer {api_key}"

		models_url = f"{base_url}/models"

Uh oh!

Conversation

RheagalFire commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Tests

Risk / Compatibility

Example usage

Uh oh!

RheagalFire commented Jun 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WMC001 commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

RheagalFire commented Jun 2, 2026 •

edited

Loading