No support for multi-GPU

[parviste-fortum]

It seems that it's not possible to run models using multiple gpus, e.g. by passing device_map="auto" to pipelines.

Is there any way to work around this limitation?

[philschmid]

What model are you trying to run? We recommend for LLMs to use the LLM container

[parviste-fortum]

Yes, I moved over to the TGI container. I had started with the generic container since it's what is used by https://registry.terraform.io/modules/philschmid/sagemaker-huggingface/aws/latest. The lack of multi-gpu support was just quite surprising, especially since it doesn't really seem to be any specific reason for not having it (I suppose no one has just implemented it).

[MoritzLaurer]

I think this blog post should respond to your question @parviste-fortum : https://www.philschmid.de/sagemaker-multi-replica