CUDA Driver not installed on the compute instance? (Docker image I guess)

[kchaitanyabandi]

Hi,

I started using the Pytorch Estimator to train an image classification network. I found that no matter what dedicated compute instance I used (4xK80, 4xP40, or 1xV100), the torch.cuda.is_available() command in the entry script always returned False.

But the funny thing is that the same command returns True as the CUDA driver is installed on the compute kernel when I checked on the notebook server in Azure ML.

And based on further digging, I saw that the compute instance itself is a windows machine with the CUDA drivers installed, but when we run the entry script using the Pytorch estimator class, the run happens in a docker image I guess that has Ubuntu 18.04 LTS running with no CUDA driver installed.

I'm a bit confused on what to do to make the CUDA work for my training purposes. Any quick help is appreciated.

Thank you.

[GiftA-MSFT]

@kchaitanyabandi we will review your feedback and get back to you shortly. Thanks.

[kchaitanyabandi]

Thank you. Any resolution would be super helpful as a great chunk of my work at the company depends on this.

[GiftA-MSFT]

@maxluk please can you help look into this issue further? Thanks.

[kchaitanyabandi]

I also observed that, once I start an experiment run and then stop the compute instance, for some reason, the Azure ML Studio still shows the experiment as running even after the compute instance has stopped. Is this normal? I was under the impression that the docker container spin up happens on the compute instance. Please correct me if I'm wrong.