how can two models be trained in distributed mode simultaneously on separate GPUs?

[fatemehtd]

Consider we want to train two models in distributed mode and each uses 2 GPUs (4 GPUs available) how do you command the second training to use 2 specific gpus (or at least not the one used by the first training)

[rwightman]

Set CUDA_VISIBLE_DEVICES in two different envs with non overlapping device ids and treat each as 2x GPU

[rwightman]

Or map different devices into different docker containers