Multiple GPU Windows System

[ptrem]

Hi,
I have a Workstation with two RTX A6000 GPUs and a Windows System 🙈
and I would like to use both GPUs with Lightning-AI.
It's possible to use just use one of the GPUs but i get the following error if I try to use both GPUs.

The training works fine for:

trainer = pl.Trainer(
        accelerator="gpu",
        devices=1,
        max_epochs=epochs,
        logger=wandb_logger)

But I can't use for example devces=2, devices=[0, 1] or devices=-1.
In every case I get the following error:
RuntimeError: Distributed package doesn't have NCCL built in

Has anyone an idea how to run Lightning on Multiple GPUs on a Windows System?
Thanks

[davidgill97]

It's bit late, but i believe nccl backend for ddp training is not supported for windows.
This is how you should configure your trainer on a windows system, but i experienced a significant slowdown before, so do compare the speed with nccl on a linux machine.

from lightning.pytorch.strategies import DDPStrategy
ddp = DDPStrategy(process_group_backend="gloo")
trainer = Trainer(strategy=ddp, accelerator="gpu", devices=2)

[ptrem]

Thanks for your reply. I will try it out.

[davidgill97]

It's bit late, but i believe nccl backend for ddp training is not supported for windows. This is how you should configure your trainer on a windows system, but i experienced a significant slowdown before, so do compare the speed with nccl on a linux machine.

from lightning.pytorch.strategies import DDPStrategy
ddp = DDPStrategy(process_group_backend="gloo")
trainer = Trainer(strategy=ddp, accelerator="gpu", devices=2)

I tried it but Trainer doesn't have parameter where you can set strategy==ddp.

I'm not sure if i understood that correctly. According to the documentation, Trainer indeed has a parameter strategy.