Is it normal that Qwen/Qwen2.5-VL-7B-Instruct takes nearly 24 GB graphic memory?

[ChenZhongPu]

I used the command vllm serve Qwen/Qwen2.5-VL-7B-Instruct , in which "torch_dtype": "bfloat16". But it shows that it takes nearly 24 GB graphic memory. Is it normal?

In my understanding, it would take only about 14GB graphic memory.

[gameofdimension]

You should factor in the KV cache

[ChenZhongPu]

Thanks. Can you please recommend some resources regarding estimating memory overhead in vllm?