Loading custom conda envs not working in SageMaker

Question

I have installed miniconda on my AWS SageMaker persistent EBS instance. Here is my starting script:

#!/bin/bash

set -e

# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
# 
# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package
# to add that as a kernel in Jupyter.
#
# For another example, see:
# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment

sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
WORKING_DIR=/home/ec2-user/SageMaker/

for env in $WORKING_DIR/miniconda/envs/*; do
    BASENAME=$(basename "$env")
    source "$WORKING_DIR/miniconda/bin/activate"
    source activate "$BASENAME"
    pip install ipykernel boto3
    python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"
done
# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.
# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py
# rm /home/ec2-user/.condarc
EOF

echo "Restarting the Jupyter server.."
restart jupyter-server

I use this in order to load my custom envs. However, when I access the JupyterLab interface, even if I see that the activated kernel is the Custom one, the only version of python running on my notebook kernel is /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python:

I also inspected the CloudWatch logs, and I see this error log: Could not find conda environment: [custom_env].

But, when I run the commands of the starting script within the JupyterLab terminal, conda succeeds in finding those envs. So the question is: what am I missing?

Thanks a lot.

Answer 1

Using !which python in a jupyter cell will always use the default system python.

But, if you selected your custom kernel in jupyter, the python used behind the scenes is the right one, you can verify it by comparing :

!python --version

!/home/ec2-user/SageMaker/miniconda/envs/<YOUR_CUSTOM_ENV_NAME> --version