Any idea to run on 8 V100-32G gpus?

[civat]

I tried:

1. offloading (optimizer state and parameters)
2. gradient checkpointing
3. reduce the max prompt length (256 now)
4. per_device_train_batch_size = 1
5. num_generations=1

OOM still occurs.
Any ideas to solve? (No A100 available now)

Thank you very much!

[Quinn777]

use lora

[m-Just]

Freezing the vision encoder saves a lot of memory.

[happyz123456789]

Could you teach me how do you solve it?

[civat]

Could you teach me how do you solve it?

Use PyTorch's torch.utils.checkpoint

Most parts of the full model do not support the gradient checkpointing even if gradient_checkpointing=True in the config file. So you need to add torch.utils.checkpoint in the expensive computation in the modeling_qwen2_vl.py file.

After that, the training memory at the first step is about 15GB, which is a huge reduction.
But there is a new issue. I found that the memory increases after several training steps, and OOM occurs at the 16-th step. I do not why. I am trying to profile the memory usage.

[happyz123456789]

Could you teach me how do you solve it?

Use PyTorch's torch.utils.checkpoint

Most parts of the full model do not support the gradient checkpointing even if gradient_checkpointing=True in the config file. So you need to add torch.utils.checkpoint in the expensive computation in the modeling_qwen2_vl.py file.

After that, the training memory at the first step is about 15GB, which is a huge reduction. But there is a new issue. I found that the memory increases after several training steps, and OOM occurs at the 16-th step. I do not why. I am trying to profile the memory usage.

thank you very much