AWS Sagemaker - Install External Library and Make it Persist

Question

I have a sagemaker instance up and running and I have a few libraries that I frequently use with it but each time I restart the instance they get wiped and I have to reinstall them. Is it possible to install my libraries to one of the anaconda environments and have the change remain?

Answer 1

The supported way to do this for Sagemaker notebook instances is with Lifecycle Configurations.

You can create an onStart lifecycle hook that can install the required packages into the respective Conda environments each time your notebook instance starts.

Please see the following blog post for more details

https://aws.amazon.com/blogs/machine-learning/customize-your-amazon-sagemaker-notebook-instances-with-lifecycle-configurations-and-the-option-to-disable-internet-access/

Answer 2

When creating you model, you can specify the requirements.txt as an environment variable.

For Eg.

env = {
    'SAGEMAKER_REQUIREMENTS': 'requirements.txt', # path relative to `source_dir` below.
}
sagemaker_model = TensorFlowModel(model_data = 's3://mybucket/modelTarFile,
                                  role = role,
                                  entry_point = 'entry.py',
                                  code_location = 's3://mybucket/runtime-code/',
                                  source_dir = 'src',
                                  env = env,
                                  name = 'model_name',
                                  sagemaker_session = sagemaker_session,
                                 )

This would ensure that the requirements file is run after the docker container is created, before running any code on it.