Use OmniParser to create a UI Surfer Agent

[ekzhu]

Confirmation

• I confirm that I am a maintainer and so can use this template. If I am not, I understand this issue will be closed and I will be asked to use a different template.

Issue body

1. UI tools for controlling with keyboards and mouse, as well as capturing screenshots. Considering a MCP server for possible implementation.
2. Use OmniParser to convert screenshots to structured messages to store in model context.

[pengjunfeng11]

Hello, has this feature been developed?? If i can help , please let me know

[ekzhu]

It's not currently being developed. You are welcome to contribute! Let's discuss the design here first before creating a PR.

[pengjunfeng11]

Before we start, the first step to do is to develop the MCP of OmniParser. The second is consider structured storage issues. The third step is how to control the mouse and keyboard. Maybe we can refer to open interface？

[ekzhu]

@yadong-lu for comment. Do you think an MCP server for OmniParser makes sense? Or it's an overkill. What's the best way to serve the model?

For message context storage it's straightforward just follow what AssistantAgent is using.

I need to look into open interface

[pengjunfeng11]

@yadong-lu for comment. Do you think an MCP server for OmniParser makes sense? Or it's an overkill. What's the best way to serve the model?

For message context storage it's straightforward just follow what AssistantAgent is using.

I need to look into open interface

My bad, the correct link should be this: https://github.com/OpenInterpreter/open-interpreter. (open-interpreter not the open interface

[yadong-lu]

@pengjunfeng11 @ekzhu thanks a lot!

• For the MCP, I think for the simplest first step we may just assume the user will provide a screenshot. It is more versatile that user may use it for various apps which they will likely build the MCP for their app.
• I think OmniTool itself can be seen as an MCP that connects GPT4o+omniparser with a windows vm (pre-installed with suite of windows app). https://github.com/microsoft/OmniParser/tree/master/omnitool
• For the control part, maybe we can leverage similar things implemented in omniparser repo: https://github.com/microsoft/OmniParser/blob/547cb8f743e3ceca030ada2c9c5eb555349fbc00/omnitool/gradio/tools/computer.py#L62