[BUG] invalid message format when using OpenAIServerModel

[sijunhe]

Describe the bug

The latest code runs into invalid message format when using OpenAIServerModel

Code to reproduce the error

from smolagents import CodeAgent, DuckDuckGoSearchTool,OpenAIServerModel

model = OpenAIServerModel(
    model_id="<...>",
    api_base="<...>",
    api_key="<...>"
)

agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")

Error logs (if any)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}
[Step 0: Duration 0.26 seconds]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 2 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Error code: 400 - {'error': {'code': 'invalid_argument', 'message': 'Message index[1] content should be string', 'type': 'invalid_request_error'}}

Expected behavior
A clear and concise description of what you expected to happen.

Packages version:
main branch

Additional context

Pretty sure this is due to OpenAI SDK uses flat message format without images but nested message format with image. The current code always use nested message, which results in this bug.

without image

client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Say this is a test",
        }
    ],
    model="gpt-4o",
)

with image

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:{img_type};base64,{img_b64_str}"},
                },
            ],
        }
    ],
)

[keruru0987]

Thank you for sharing this!

[aymeric-roucher]

@sijunhe this works fine for me with 1.9.1:

from smolagents import CodeAgent, DuckDuckGoSearchTool,OpenAIServerModel

model = OpenAIServerModel(
    model_id="gpt-4",
)

agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=model)

agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")

[aymeric-roucher]

So could you precise which model you're using?