Distributed Training Env Variables

[ademyanchuk]

Thank you for the brilliant work you are doing. I have a question with regard to DDP training. What environment variables should one setup before start training?
Thanks in advance!

Cheers,
Alexey

[rwightman]

@ademyanchuk I don't set any, I use the distributed_train.sh script as is for same-node DDP training.

For multi-node, on the rare occasions I've used it, I manually add the rank/master info into the args of a similar shell script per-machine like the example below.

python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_PER_MACHINE
               --nnodes=2 --node_rank=0 --master_addr="192.168.1.1"
               --master_port=1234 train.py 

[ademyanchuk]

Thank you, @rwightman !