[Feature] DSPy Audio/Video Support Tracking

[isaacbmiller]

What feature would you like to see?

We have received a number of requests for Audio and Video input support over the last few months (#2037, #7844, etc.)

I implemented DSPy.Image, and am looking for someone to help out and create similar or better implementations for audio and/or video inputs. It would be shocking to me if some good prompting and few shot suppport for audio would greatly help in some use cases, and also being able to script with audio in the same way that you can with text inputs.

For someone to implement this, there are a few required steps for the implementation I am imagining:

1. create a class similar to Image (see adapters/image_utils.py)
2. edit chat_adapter and json_adapter to have a try_expand_audio_tags method that will search and expand messages with multimodal inputs
3. Write tests similar to tests/signatures/test_adapter_image.py to make sure it can work with a variety of signature types and input methods

I don't know much about the audio input APIs to really know what the speedbumps on this implementation are going to be.

As a first step, I would choose either the OpenAI API or Gemini, and get it working for that provider with whatever hacky code is needed, then expand and abstract after that.

feel free to @ me in the discord username is ibmiller if you need help

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

No response

[pretbc]

Yes, I'd like to help implement this.

[ramisbahi]

Yes, I'd like to help implement this: I'd like to help with implementing support for Video input. Additionally, I'm looking into benchmarking video understanding performance to evaluate how well DSPy can process video-based inputs