Kernel dying when trying to train model on aws sagemaker

[shubham-scisar]

I am trying to train a batch size of 30 images of size 512*512 using ml.p2.xlarge instance on conda_amazonei_tensorflow_p36 notebook. The kernel dies without giving any error. I am training utilising keras-unet package available through pip.

[icywang86rui]

@shubhamsharma1609
Sorry that you run into this issue.
Could you show us your code which used to start the training job?
Did you use the python sdk? If so which version?

ml.p2.xlarge instance on conda_amazonei_tensorflow_p36 notebook

Did you mean the training job was launched from the SageMaker hosted notebook instance.
Did the kernel die when you run a specific line of code?
Have you tried to restart the kernel?

[shubham-scisar]

I did not use sagemaker pythonsdk. I installed keras-unet and with higher batch size i.e. 30 ,I don't get any error but the kernel dies. I am utilising the compute power of AWS sagemaker and nothing else from the sdk.

history = model.fit_generator(
    train_gen,
    steps_per_epoch=50,
    epochs=20,
    
    validation_data=(x_val, y_val),
    callbacks=[callback_checkpoint]
)

[shubham-scisar]

I have somehow solved the problem by reducing the batch size, now my notebook is running for each epoch. I just want to ask that the Jupyter session is of 12 hours but my training job will take more than that, Can you please guide how can I increase the Jupyter session duration, so that it does not stop training. Also, can I close my browser but do not stop the instance and at a later time connect to the jupyter notebook and check the training is happening continuously ?