How to install smdistributed ? (I wanto to use custom docker image)

[hisashi-ito]

I want to try smdistributed training using custom docker image (not tf, pythorch image) on ECR or local instance (local mode).
This is related to this question, and I get the following error as well. What would you suggest I do?

#2452

• start script

from sagemaker.estimator import Estimator
estimator = Estimator(
    image_uri=IMAGE_NAME,
    role=ROLE,
    train_instance_type="local", 
    train_instance_count=1,
    hyperparameters={},
    distribution={"smdistributed": {"dataparallel": {"enabled": True}}}
)

• result

train_instance_count has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
train_instance_type has been renamed in sagemaker>=2.
See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.
Creating icwyyk0vyo-algo-1-zssqf ...
Creating icwyyk0vyo-algo-1-zssqf ... done
Attaching to icwyyk0vyo-algo-1-zssqf
icwyyk0vyo-algo-1-zssqf | 2021-12-27 12:56:23,368 sagemaker-training-toolkit INFO     Invoking user script
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | Training Env:
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | {
icwyyk0vyo-algo-1-zssqf |     "additional_framework_parameters": {},
icwyyk0vyo-algo-1-zssqf |     "channel_input_dirs": {},
icwyyk0vyo-algo-1-zssqf |     "current_host": "algo-1-zssqf",
icwyyk0vyo-algo-1-zssqf |     "framework_module": null,
icwyyk0vyo-algo-1-zssqf |     "hosts": [
icwyyk0vyo-algo-1-zssqf |         "algo-1-zssqf"
icwyyk0vyo-algo-1-zssqf |     ],
icwyyk0vyo-algo-1-zssqf |     "hyperparameters": {},
icwyyk0vyo-algo-1-zssqf |     "input_config_dir": "/opt/ml/input/config",
icwyyk0vyo-algo-1-zssqf |     "input_data_config": {},
icwyyk0vyo-algo-1-zssqf |     "input_dir": "/opt/ml/input",
icwyyk0vyo-algo-1-zssqf |     "is_master": true,
icwyyk0vyo-algo-1-zssqf |     "job_name": "sagemaker-example-2021-12-27-12-56-16-061",
icwyyk0vyo-algo-1-zssqf |     "log_level": 20,
icwyyk0vyo-algo-1-zssqf |     "master_hostname": "algo-1-zssqf",
icwyyk0vyo-algo-1-zssqf |     "model_dir": "/opt/ml/model",
icwyyk0vyo-algo-1-zssqf |     "module_dir": "/opt/ml/code",
icwyyk0vyo-algo-1-zssqf |     "module_name": "train",
icwyyk0vyo-algo-1-zssqf |     "network_interface_name": "eth0",
icwyyk0vyo-algo-1-zssqf |     "num_cpus": 8,
icwyyk0vyo-algo-1-zssqf |     "num_gpus": 1,
icwyyk0vyo-algo-1-zssqf |     "output_data_dir": "/opt/ml/output/data",
icwyyk0vyo-algo-1-zssqf |     "output_dir": "/opt/ml/output",
icwyyk0vyo-algo-1-zssqf |     "output_intermediate_dir": "/opt/ml/output/intermediate",
icwyyk0vyo-algo-1-zssqf |     "resource_config": {
icwyyk0vyo-algo-1-zssqf |         "current_host": "algo-1-zssqf",
icwyyk0vyo-algo-1-zssqf |         "hosts": [
icwyyk0vyo-algo-1-zssqf |             "algo-1-zssqf"
icwyyk0vyo-algo-1-zssqf |         ]
icwyyk0vyo-algo-1-zssqf |     },
icwyyk0vyo-algo-1-zssqf |     "user_entry_point": "train.py"
icwyyk0vyo-algo-1-zssqf | }
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | Environment variables:
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | SM_HOSTS=["algo-1-zssqf"]
icwyyk0vyo-algo-1-zssqf | SM_NETWORK_INTERFACE_NAME=eth0
icwyyk0vyo-algo-1-zssqf | SM_HPS={}
icwyyk0vyo-algo-1-zssqf | SM_USER_ENTRY_POINT=train.py
icwyyk0vyo-algo-1-zssqf | SM_FRAMEWORK_PARAMS={}
icwyyk0vyo-algo-1-zssqf | SM_RESOURCE_CONFIG={"current_host":"algo-1-zssqf","hosts":["algo-1-zssqf"]}
icwyyk0vyo-algo-1-zssqf | SM_INPUT_DATA_CONFIG={}
icwyyk0vyo-algo-1-zssqf | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
icwyyk0vyo-algo-1-zssqf | SM_CHANNELS=[]
icwyyk0vyo-algo-1-zssqf | SM_CURRENT_HOST=algo-1-zssqf
icwyyk0vyo-algo-1-zssqf | SM_MODULE_NAME=train
icwyyk0vyo-algo-1-zssqf | SM_LOG_LEVEL=20
icwyyk0vyo-algo-1-zssqf | SM_FRAMEWORK_MODULE=
icwyyk0vyo-algo-1-zssqf | SM_INPUT_DIR=/opt/ml/input
icwyyk0vyo-algo-1-zssqf | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
icwyyk0vyo-algo-1-zssqf | SM_OUTPUT_DIR=/opt/ml/output
icwyyk0vyo-algo-1-zssqf | SM_NUM_CPUS=8
icwyyk0vyo-algo-1-zssqf | SM_NUM_GPUS=1
icwyyk0vyo-algo-1-zssqf | SM_MODEL_DIR=/opt/ml/model
icwyyk0vyo-algo-1-zssqf | SM_MODULE_DIR=/opt/ml/code
icwyyk0vyo-algo-1-zssqf | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{},"current_host":"algo-1-zssqf","framework_module":null,"hosts":["algo-1-zssqf"],"hyperparameters":          {},"input_config_dir":"/opt/ml/input/config","input_data_config":{},"input_dir":"/opt/ml/input","is_master":true,"job_name":"sagemaker-example-2021-12-27-12-56-16-061","log_level":20,"master_hostname":          "algo-1-zssqf","model_dir":"/opt/ml/model","module_dir":"/opt/ml/code","module_name":"train","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_di          r":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-zssqf","hosts":["algo-1-zssqf"]},"user_entry_point":"train.py"}
icwyyk0vyo-algo-1-zssqf | SM_USER_ARGS=[]
icwyyk0vyo-algo-1-zssqf | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
icwyyk0vyo-algo-1-zssqf | PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | Invoking script with the following command:
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | /opt/conda/bin/python train.py
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf |
icwyyk0vyo-algo-1-zssqf | Traceback (most recent call last):
icwyyk0vyo-algo-1-zssqf |   File "train.py", line 30, in <module>
icwyyk0vyo-algo-1-zssqf |     import smdistributed.dataparallel.torch.distributed as dist
icwyyk0vyo-algo-1-zssqf | ModuleNotFoundError: No module named 'smdistributed'
icwyyk0vyo-algo-1-zssqf | 2021-12-27 12:56:24,452 sagemaker-training-toolkit ERROR    Reporting training FAILURE
icwyyk0vyo-algo-1-zssqf | 2021-12-27 12:56:24,452 sagemaker-training-toolkit ERROR    ExecuteUserScriptError:
icwyyk0vyo-algo-1-zssqf | ExitCode 1
icwyyk0vyo-algo-1-zssqf | ErrorMessage ""
icwyyk0vyo-algo-1-zssqf | Command "/opt/conda/bin/python train.py"
icwyyk0vyo-algo-1-zssqf | 2021-12-27 12:56:24,452 sagemaker-training-toolkit ERROR    Encountered exit_code 1
icwyyk0vyo-algo-1-zssqf exited with code 1

[pnipinto]

any updates on this issue? i'm having the same problem with pytorch 2.1.0

[rohit901]

same issue, how to use it with HuggingFace container? getting module not found error for smdistributed