MultiModal model batched inference

[aswad546]

Hello all,

I am using a fine tuned version of Qwen 2VL for a multimodal LLM application and I need to make many queries to the model at the same time. I was wondering if VLLM provides dynamic batching for multimodal inputs out of the box or if there is something special I have to do to make sure this is the case. If someone could point me in the right direction I would be very grateful.

Thank you!

[AlexisMDP]

@aswad546 did you found a way to do this? I'm looking for the exact same thing. Thanks!

[aswad546]

Hey, VLLM does support dynamic batching automatically so you don't need to do much just serve the model and send concurrent queries it will auto batch them and run them through the model.

[AlexisMDP]

Thanks!