feat: enhance model validation with detailed error messages and diagnostics #1155

Angel98518 · 2026-02-04T21:53:32Z

Fixes #231

This PR significantly enhances the model validation system to provide detailed error messages and diagnostic information, helping users understand the root cause of validation failures.

Changes

Enhanced Model Validation (`backend/app/component/model_validation.py`)

Added ValidationStage enum to track validation stages (initialization, model creation, agent creation, model call, tool call execution, response parsing)
Added ValidationErrorType enum for error categorization (authentication, network, model not found, rate limits, quota, timeouts, tool call support, configuration errors)
Added ValidationResult class with comprehensive diagnostic information
Added categorize_error() function to classify errors by type
Added validate_model_with_details() function with step-by-step validation:
1. Initialization validation
2. Model creation validation
3. Agent creation validation
4. Model call execution validation
5. Tool call execution validation
Each stage provides detailed error messages and diagnostic information
Error categorization covers all common failure scenarios

Enhanced Controller (`backend/app/controller/model_controller.py`)

Updated to use new detailed validation function
Added include_diagnostics parameter for optional detailed diagnostics
Enhanced response model with:
- error_type: Specific error category
- failed_stage: Stage where validation failed
- successful_stages: List of stages that succeeded
- diagnostic_info: Detailed diagnostic information
Improved error messages based on error type and stage
Better logging with diagnostic context

Benefits

✅ Better Error Messages: Users get specific, actionable error messages instead of generic failures
✅ Root Cause Identification: Clear indication of which stage failed (initialization, model creation, agent creation, model call, tool call)
✅ Diagnostic Information: Optional detailed diagnostics help debug configuration issues
✅ Error Categorization: Errors are categorized (authentication, network, quota, etc.) for better handling
✅ Backward Compatible: Maintains existing API while adding optional diagnostic features

Example Error Messages

Authentication: "Authentication failed. Please check your API key and ensure it is valid and has the necessary permissions."
Model Not Found: "Model 'X' not found on platform 'Y'. Please verify the model name and platform are correct."
Network Error: "Network error occurred. Please check your internet connection and try again."
Tool Call Not Supported: "Model call succeeded, but this model does not support tool calling functionality. Please try with another model that supports tool calls."
Tool Call Execution Failed: "Tool call was made but execution failed. Expected result: '...', but got: '...'"

All error messages provide actionable guidance to help users resolve issues.

Code Statistics

Lines Added: 547 insertions
Files Modified: 2 files
New Classes: 3 (ValidationStage, ValidationErrorType, ValidationResult)
New Functions: 2 (categorize_error, validate_model_with_details)

…ostics Fixes eigent-ai#231 This PR significantly enhances the model validation system to provide detailed error messages and diagnostic information, helping users understand the root cause of validation failures. - Added enum to track validation stages - Added enum for error categorization - Added class with comprehensive diagnostic information - Added function to classify errors by type - Added function with step-by-step validation: 1. Initialization validation 2. Model creation validation 3. Agent creation validation 4. Model call execution validation 5. Tool call execution validation - Each stage provides detailed error messages and diagnostic information - Error categorization covers: authentication, network, model not found, rate limits, quota, timeouts, tool call support, and configuration errors - Updated to use new detailed validation function - Added parameter for optional detailed diagnostics - Enhanced response model with: - : Specific error category - : Stage where validation failed - : List of stages that succeeded - : Detailed diagnostic information - Improved error messages based on error type and stage - Better logging with diagnostic context - **Better Error Messages**: Users get specific, actionable error messages instead of generic failures - **Root Cause Identification**: Clear indication of which stage failed (initialization, model creation, agent creation, model call, tool call) - **Diagnostic Information**: Optional detailed diagnostics help debug configuration issues - **Error Categorization**: Errors are categorized (authentication, network, quota, etc.) for better handling - **Backward Compatible**: Maintains existing API while adding optional diagnostic features - Authentication: "Authentication failed. Please check your API key..." - Model Not Found: "Model 'X' not found on platform 'Y'..." - Network Error: "Network error occurred. Please check your connection..." - Tool Call Not Supported: "Model call succeeded, but this model does not support tool calling functionality..." - Tool Call Execution Failed: "Tool call was made but execution failed..." All error messages provide actionable guidance to help users resolve issues.

4pmtong · 2026-02-05T07:22:08Z

Thanks @Angel98518 for the contribution! @bytecii could you help with the review?

bytecii · 2026-02-06T05:46:25Z

@4pmtong This PR is dependent on #1017
where we should raise the error to break stop the task if the model has any errors

4pmtong · 2026-02-06T08:41:48Z

@4pmtong This PR is dependent on #1017 where we should raise the error to break stop the task if the model has any errors

Hi @bytecii Thanks for the feedback. After reviewing both PRs:
Dependency: #1017 depends on #1155. #1155 provides validate_model_with_details and ValidationResult, which #1017 can reuse.
Different use cases: #1155 handles Settings validation (returns 200 with structured results for the frontend). #1017 handles pre-task validation and should raise to stop the task when the model is invalid.
Recommendation: #1017 can build on #1155 by reusing its validation logic and adding the raise behavior in the task flow.

4pmtong · 2026-02-06T08:55:02Z

backend/app/component/model_validation.py

+    if any(
+        keyword in error_str
+        for keyword in [
+            "invalid",
+            "configuration",
+            "config",
+            "parameter",
+            "param",
+        ]
+    ):
+        return ValidationErrorType.INVALID_CONFIGURATION


Hi @Angel98518 The "invalid" keyword is too broad and can misclassify errors like "Invalid request" as INVALID_CONFIGURATION. Consider narrowing it to more specific phrases

4pmtong · 2026-02-06T08:58:46Z

backend/app/component/model_validation.py

@@ -28,6 +103,111 @@ def get_website_content(url: str) -> str:
    return "Tool execution completed successfully for https://www.camel-ai.org, Website Content: Welcome to CAMEL AI!"


Hi @Angel98518 Consider extracting it to a constant, e.g. EXPECTED_TOOL_RESULT, to avoid duplication and simplify future changes.

bytecii · 2026-02-06T09:02:55Z

backend/app/component/model_validation.py

+    if any(
+        keyword in error_str
+        for keyword in [
+            "invalid_api_key",


I am not sure whether these keywords checking are robust enough. Because for now eigent supports a lot of different model platform / providers, such as litellm, qwen etc.

- Narrow 'invalid' keyword to specific configuration error phrases to avoid false positives - Extract EXPECTED_TOOL_RESULT constant to avoid duplication - Enhance error categorization for different model platforms/providers (OpenAI, Anthropic, LiteLLM, Qwen, etc.) - Improve error detection by checking exception types first, then error messages - Add more comprehensive keyword lists for better error classification across providers

Angel98518 · 2026-02-06T16:15:32Z

Hi, @4pmtong I have fixed all feedbacks

Wendong-Fan · 2026-02-06T17:59:45Z

Thanks for working on this! The staged validation approach looks solid. Just wanted to share some thoughts on the error classification part.

The string matching for error categorization (checking for keywords like "401", "authentication", etc.) might be fragile since different AI providers format their errors differently, and they could change the wording anytime.

Here's the thing though: error messages from OpenAI, Anthropic and other major providers are already pretty clear, like:

Error code: 401 - Invalid API key provided

Users can understand what's wrong just by reading this. Adding our own "translation" layer might actually lose useful context.

A simpler approach could be:

def format_validation_error(exception: Exception) -> str:
    raw = str(exception)
    if len(raw) > 300:
        raw = raw[:300] + "..."
    return raw

Less code, fewer edge cases. When validation fails, showing the raw error is often good enough, WDYT?

cc @4pmtong @bytecii

- Always preserve raw error messages from providers (they are usually clear) - Make error categorization more conservative - only categorize when confident (exception types, HTTP status codes, ModelProcessingError) - Remove fragile keyword-based string matching that could misclassify errors - Use raw error messages as primary messages instead of translating them - Add format_raw_error() helper function to truncate long errors - Add raw_error_message field to ValidationResult for preserving original errors This approach: - Preserves useful context from provider error messages - Reduces risk of misclassification - Still provides error categorization when we have high confidence - Maintains backward compatibility with error_type field

Angel98518 · 2026-02-09T00:42:23Z

Hi, @Wendong-Fan , could you review my changes again please?

backend/app/component/model_validation.py

backend/app/controller/model_controller.py

backend/app/component/model_validation.py

- Add missing fields (model_response_info, tool_call_info, validation_stages) to ValidateModelResponse - Reduce step_timeout from 1800s to 60s for validation (more reasonable timeout) - Remove meaningless tools_count from agent_creation diagnostic info

- Remove traceback from error_details (security: don't expose internal info to clients) - Move is_valid=True to after stage 5 (TOOL_CALL_EXECUTION) instead of stage 4 - Add missing fields to ValidateModelResponse (model_response_info, tool_call_info, validation_stages) - Reduce step_timeout from 1800s to 60s for validation - Remove meaningless tools_count from agent_creation diagnostic info

Angel98518 · 2026-02-09T15:28:48Z

Hi, @Zephyroam , I have fixed feedbacks, please check. Thank you for your review.

bytecii

Also let's run the pre-commit for the changed files.
and add the files you changed / added to the

eigent/.github/workflows/pre-commit.yml

Line 34 in 0e6847d

uv run pre-commit run --files \

bytecii · 2026-02-10T07:04:32Z

backend/app/controller/model_controller.py

+
+        result = ValidateModelResponse(**response_data)
+
+        logger.info(


Can we change to error or warning if there is anything wrong?

bytecii · 2026-02-10T07:05:50Z

backend/app/controller/model_controller.py

Let's add tests to for example tests/app/component/test_model_validation.py and tests/app/controller/test_model_controller.py

Angel98518 force-pushed the feat/enhance-model-validation-messages branch from 37e6ab0 to 4ae05b6 Compare February 4, 2026 22:00

Angel98518 added 2 commits February 5, 2026 00:02

fix: apply pre-commit fixes (trailing whitespace and ruff format)

4f69a95

fix: remove unused traceback_str variable

b189352

4pmtong requested review from 4pmtong, bytecii and fengju0213 February 5, 2026 07:22

Wendong-Fan requested a review from Zephyroam February 5, 2026 22:33

Wendong-Fan added this to the Sprint 13 milestone Feb 5, 2026

4pmtong mentioned this pull request Feb 6, 2026

fix: fix task nonstopping error if model key is expired #1017

Closed

4 tasks

4pmtong reviewed Feb 6, 2026

View reviewed changes

bytecii reviewed Feb 6, 2026

View reviewed changes

Angel98518 force-pushed the feat/enhance-model-validation-messages branch from 14ad3e0 to 7add156 Compare February 6, 2026 19:28

Angel98518 requested review from 4pmtong and bytecii February 8, 2026 21:17

Zephyroam reviewed Feb 9, 2026

View reviewed changes

Angel98518 added 2 commits February 9, 2026 16:20

Angel98518 requested a review from Zephyroam February 9, 2026 15:51

Merge branch 'main' into feat/enhance-model-validation-messages

b58abda

bytecii reviewed Feb 10, 2026

View reviewed changes

		@@ -28,6 +103,111 @@ def get_website_content(url: str) -> str:
		return "Tool execution completed successfully for https://www.camel-ai.org, Website Content: Welcome to CAMEL AI!"

feat: enhance model validation with detailed error messages and diagnostics #1155

Are you sure you want to change the base?

feat: enhance model validation with detailed error messages and diagnostics #1155

Conversation

Angel98518 commented Feb 4, 2026

Changes

Enhanced Model Validation (backend/app/component/model_validation.py)

Enhanced Controller (backend/app/controller/model_controller.py)

Benefits

Example Error Messages

Code Statistics

Uh oh!

4pmtong commented Feb 5, 2026

Uh oh!

bytecii commented Feb 6, 2026

Uh oh!

4pmtong commented Feb 6, 2026

Uh oh!

4pmtong Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

4pmtong Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

bytecii Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Angel98518 commented Feb 6, 2026

Uh oh!

Wendong-Fan commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Angel98518 commented Feb 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Angel98518 commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bytecii left a comment

Choose a reason for hiding this comment

Uh oh!

bytecii Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

bytecii Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Enhanced Model Validation (`backend/app/component/model_validation.py`)

Enhanced Controller (`backend/app/controller/model_controller.py`)

Wendong-Fan commented Feb 6, 2026 •

edited

Loading

Angel98518 commented Feb 9, 2026 •

edited

Loading