MultimodalWebSurfer Agent fails with multiple errors when trying to search restaurants

[qiqifeng]

What happened?

When trying to use MultimodalWebSurfer agent to search for restaurants, the following cascade of errors occurred:

Initial connection error when trying to access Bing:
playwright._impl._errors.Error: Page.goto: net::ERR_CONNECTION_RESET at https://www.bing.com/
Followed by httpcore.ReadError and httpx.ReadError

Finally resulting in:
openai.UnprocessableEntityError: Failed to deserialize the JSON body into the target type: messages[3]: data did not match any variant of untagged enum ChatCompletionRequestContent
Steps to Reproduce

1. Created a MultimodalWebSurfer agent with DeepSeek chat model
2. Attempted to run a simple restaurant search query
3. Used RoundRobinGroupChat with max_turns=3
   `model_client = OpenAIChatCompletionClient(
   model="deepseek-chat",
   base_url="https://api.deepseek.com",
   api_key="xxx",
   model_capabilities={
   "vision": True,
   "function_calling": True,
   "json_output": True,
   },
   )

web_surfer_agent = MultimodalWebSurfer(
name="MultimodalWebSurfer",
model_client=model_client,
)

agent_team = RoundRobinGroupChat([web_surfer_agent], max_turns=3)`

Actual Behavior
Multiple errors occur preventing the agent from performing the search:

Cannot connect to Bing
API connection errors
JSON deserialization errors
Questions
Is this a known issue with the MultimodalWebSurfer agent?
Could this be related to incompatibility between DeepSeek chat model and the web surfing capabilities?
Are there any workarounds or configuration changes needed?

What did you expect to happen?

The agent should be able to search and provide restaurant recommendations using web browsing capabilities.

How can we reproduce it (as minimally and precisely as possible)?

`import asyncio
from autogen_agentchat.ui import Console
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.agents.web_surfer import MultimodalWebSurfer

async def main() -> None:
model_client = OpenAIChatCompletionClient(
model="deepseek-chat",
base_url="https://api.deepseek.com",
api_key="sk-b924227d833a4e71bc4a56e1b02fef90",
model_capabilities={
"vision": True,
"function_calling": True,
"json_output": True,
},
)
# Define an agent
web_surfer_agent = MultimodalWebSurfer(
name="MultimodalWebSurfer",
model_client=model_client,
)

# Define a team
agent_team = RoundRobinGroupChat([web_surfer_agent], max_turns=3)

# Run the team and stream messages to the console
stream = agent_team.run_stream(task="try to help me find a good restaurant in new york.")
await Console(stream)
# Close the browser controlled by the agent
await web_surfer_agent.close()

asyncio.run(main())`

error message
`PS F:\ai_study> f:; cd 'f:\ai_study'; & 'c:\Users\11481\anaconda3\envs\autogen\python.exe' 'c:\Users\11481.cursor\extensions\ms-python.debugpy-2024.6.0-win32-x64\bundled\libs\debugpy\adapter/../..\debugpy\launcher' '63710' '--' 'F:\ai_study\test.py'
c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\models\openai_openai_client.py:1020: DeprecationWarning: model_capabilities is deprecated, use model_info instead
super().init(
---------- user ----------
try to help me find a good restaurant in new york.
---------- MultimodalWebSurfer ----------
Web surfing error:

Traceback (most recent call last):
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 383, in on_messages_stream
content = await self._generate_reply(cancellation_token=cancellation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 410, in _generate_reply
await self._lazy_init()
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 287, in _lazy_init
await self._page.goto(self.start_page)
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\playwright\async_api_generated.py", line 8973, in goto
await self._impl_obj.goto(
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\playwright_impl_page.py", line 551, in goto
return await self._main_frame.goto(**locals_to_params(locals()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\playwright_impl_frame.py", line 145, in goto
await self._channel.send("goto", locals_to_params(locals()))
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\playwright_impl_connection.py", line 61, in send
return await self._connection.wrap_api_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\playwright_impl_connection.py", line 528, in wrap_api_call
raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.Error: Page.goto: net::ERR_CONNECTION_RESET at https://www.bing.com/
Call log:

• navigating to "https://www.bing.com/", waiting until "load"

---------- MultimodalWebSurfer ----------
Web surfing error:

Traceback (most recent call last):
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_transports\default.py", line 101, in map_httpcore_exceptions
yield
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_transports\default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\connection_pool.py", line 256, in handle_async_request
raise exc from None
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\http_proxy.py", line 343, in handle_async_request
return await self._connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\http11.py", line 136, in handle_async_request
raise exc
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\http11.py", line 106, in handle_async_request
) = await self._receive_response_headers(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\http11.py", line 177, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_async\http11.py", line 217, in _receive_event
data = await self._network_stream.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_backends\anyio.py", line 32, in read
with map_exceptions(exc_map):
^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\contextlib.py", line 158, in exit
self.gen.throw(value)
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpcore_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1582, in _request
response = await self._client.send(
^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_transports\default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\contextlib.py", line 158, in exit
self.gen.throw(value)
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\httpx_transports\default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 383, in on_messages_stream
content = await self._generate_reply(cancellation_token=cancellation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 509, in _generate_reply
response = await self._model_client.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\models\openai_openai_client.py", line 510, in create
result: Union[ParsedChatCompletion[BaseModel], ChatCompletion] = await future
^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai\resources\chat\completions.py", line 1702, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1849, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1543, in request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1606, in _request
return await self._retry_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1676, in _retry_request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1606, in _request
return await self._retry_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1676, in _retry_request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1616, in _request
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

---------- MultimodalWebSurfer ----------
Web surfing error:

Traceback (most recent call last):
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 383, in on_messages_stream
content = await self._generate_reply(cancellation_token=cancellation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\agents\web_surfer_multimodal_web_surfer.py", line 509, in _generate_reply
response = await self._model_client.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\autogen_ext\models\openai_openai_client.py", line 510, in create
result: Union[ParsedChatCompletion[BaseModel], ChatCompletion] = await future
^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai\resources\chat\completions.py", line 1702, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1849, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1543, in request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "c:\Users\11481\anaconda3\envs\autogen\Lib\site-packages\openai_base_client.py", line 1644, in _request
raise self._make_status_error_from_response(err.response) from None
openai.UnprocessableEntityError: Failed to deserialize the JSON body into the target type: messages[3]: data did not match any variant of untagged enum ChatCompletionRequestContent at line 1 column 1168160

sys:1: ResourceWarning: unclosed <socket.socket fd=824, family=2, type=1, proto=6, laddr=('127.0.0.1', 63871), raddr=('127.0.0.1', 7890)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
c:\Users\11481\anaconda3\envs\autogen\Lib\asyncio\proactor_events.py:116: ResourceWarning: unclosed transport <_ProactorSocketTransport fd=-1 read=<_OverlappedFuture cancelled>>
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback`

AutoGen version

0.4.3

Which package was this bug in

Core

Model used

deepseek v3

Python version

python 3.12

Operating system

windows

Any additional info you think would be helpful for fixing this bug

The errors seem to cascade from a network connection issue to API communication problems. The final error suggests there might be an issue with the message format being sent to the model.