Chart insight fails with unhelpful "LLM_UNKNOWN_ERROR" when model does not support image inputs

[KServe-FMS]

Chart insight fails with unhelpful "LLM_UNKNOWN_ERROR" when model doesn't support image inputs

Description

When generating a chart insight on a non-vision LLM (e.g. deepseek-v4-flash via OpenCode Go), the UI shows a generic Model request failed error instead of a meaningful message like "The selected model does not support image inputs. Please use a vision-capable model for chart insights."

The root cause is twofold:

1. ChartInsightAgent unconditionally sends the chart as a base64 PNG image to every model (agent_chart_insight.py:80-92). For text-only models this is an invalid request.

2. The image-deserialization fallback in client_utils.py only recognises a narrow set of provider error messages. It checks for:

   • "image_url" in lowered and "expected \text`" in lowered`
   • "unknown variant \image_url`" in lowered`

   Providers that give a different error shape — such as OpenCode Go returning 400: Error from provider (OpenCode Go): Upstream request failed — are never matched, so the image-strip-and-retry logic never fires. The error bubbles up to the UI as LLM_UNKNOWN_ERROR.

Steps to Reproduce

1. Launch Data Formulator with a text-only model (e.g. deepseek-v4-flash via OpenCode Go or any OpenAI-compatible proxy)
2. Import a dataset and create a chart (e.g. a line chart)
3. Click "Generate Insight" on the chart
4. Observe: generic Model request failed error with no actionable information

Expected Behaviour

Either:

• A) Detect the text-only model before sending the request and either:

  • Strip the image from the prompt and generate insight from metadata alone (graceful degradation), or
  • Show a clear UI message: "This model does not support image analysis. Switch to a vision-capable model (e.g. GPT-4o, Claude 3.5 Sonnet) or the insight will use chart metadata only."

• B) Expand _is_image_deserialize_error() in client_utils.py to recognise a broader set of provider error strings (e.g. "does not support image", "upstream request failed" when paired with image_url content) so the existing image-strip-and-retry fallback activates for more providers.

Actual Behaviour

UI shows Model request failed (LLM_UNKNOWN_ERROR) with no way for the user to understand why or what to change.

Relevant Code

• py-src/data_formulator/agents/agent_chart_insight.py:80-92 — always sends image_url content block
• py-src/data_formulator/agents/client_utils.py:85-88 — narrow error pattern matching in _is_image_deserialize_error
• py-src/data_formulator/agents/client_utils.py:157-162 — fallback that strips images and retries (never reached for non-matching errors)

Suggested Fix

Short term: Expand the error patterns in _is_image_deserialize_error() to catch more providers. At minimum:

def _is_image_deserialize_error(self, error_text: str) -> bool:
    lowered = error_text.lower()
    return (
        ("image_url" in lowered and "expected `text`" in lowered)
        or "unknown variant `image_url`" in lowered
        # Catch providers that fail silently on image inputs
        or ("upstream request failed" in lowered and self._has_image_in_request)
    )

But this requires tracking whether the current request actually contains images.

Better fix: Have ChartInsightAgent check whether the model supports vision before including the image, or gracefully retry without the image on any 400 error that mentions the upstream failing.

Long term: The chart insight could degrade gracefully: send metadata-only to text models (works fine, just less accurate) and include the image when a vision model is detected.

Environment

• Data Formulator version: latest (main branch, commit 70f5f8d / 3d3004c)
• Model: deepseek-v4-flash via OpenCode Go (OpenAI-compatible endpoint at https://opencode.ai/zen/go/v1)
• Reproduced on macOS