How to run Jupyter Notebook with a different version of Python?

Question

I want to be able to run both Python 3.8 (currrent version) and Python 3.7 in my Jupyter Notebook. I understand creating different IPython kernels from virtual environments is the way. So I downloaded Python 3.7 and locally installed it in my home directory. Used this python binary file to create a virtual environment by

> virtualenv -p ~/Python3.7/bin/python3 py37
> source py37/bin/activate

This works perfectly and gives 'Python 3.7' correctly on checking with python --version and sys.version. Then for creating IPython kernel,

(py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
(py37) > jupyter notebook

This also runs without error and the kernel can be confirmed to be added in the Notebook. However it does not run Python 3.7 like the virtual environment, but Python 3.8 like the default kernel. (confirmed with sys.version)

I checked ~/.local/share/jupyter/kernels/py37/kernel.json and saw its contents as

{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3.7",
 "language": "python"

So naturally I tried editing the /usr/bin/python3 to point to my Python 3.7 binary file path that is ~/Python3.7/bin/python3, but then even the kernel doesn't work properly in the notebook.

What can I possibly do?

NB: I use Arch Linux, so I installed jupyter, virtualenv, ... through pacman not pip as its recommended in Arch.

Answer 1

Found it myself, the hard way. Let me share anyway, in case this helps anyone.

I guess, the problem was that, jupyter notebook installed through pacman searches for python binary files in the PATH variable and not in the path specified by the virtual environment. Since I installed Python 3.7 locally in my home directory, Jupyter can't find it and it might have defaulted to the default python version.

So the possible solutions are:

1. Install Jupyter Notebook through pip (instead of pacman) within the virtual environment set on Python 3.7 (This is not at all recommended for Arch Linux users, as installing packages through pip can probably cause issues in future)

 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37     
 > make     
 > make install   

 > virtualenv -p ~/Python3.7/bin/python3 py37   
 > source py37/bin/activate 

 (py37) > pip install notebook   
 (py37) > python -m notebook

2. Install Python 3.7 within default directory (instead of specifying somewhere else). Create a new IPython kernel using the suitable virtual environment and use jupyter-notebook installed through pacman. (Recommended for Arch Linux users)
   Note 1: > python points to the updated global Python 3.8 version and > python3 or > python3.7 points to newly installed Python 3.7
   Note 2: Once the required kernel is created, you might even be able to use that python version outside the virtual environment.

 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure   
 > make     
 > sudo make install   

 > virtualenv -p $(which python3.7) py37   
 > source py37/bin/activate 

 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook

3. Add the path of the directory where you have locally installed the new Python version to the $PATH variable, create an IPython kernel and run Jupyter Notebook within suitable virtual environment. (Haven't yet tried this one personally. Just felt that this should work. So no guarantee. Also I don't think this is a good solution)

 > wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz  
 > tar -xvf Python-3.7.4.tgz    
 > cd Python-3.5.1/     
 > ./configure --prefix=$HOME/Python37 
 > make     
 > make install   
 > export PATH="$HOME/Python37/bin:$PATH"          

 > virtualenv -p  py37   
 > source py37/bin/activate 

 (py37) > ipython kernel install --user --name py37 --display-name "Python 3.7"
 (py37) > jupyter notebook

Answer 2

Another approach is to just run the notebook application directly using the version of Python you require - provided it's installed for that version of Python (e.g. on a Mac with brew installed version of Python3.8):

/usr/local/opt/[email protected]/bin/python3 -m notebook

Also if you want to install packages for that version:

/usr/local/opt/[email protected]/bin/pip3 install that_package

Answer 3

in the official jupyter dcoumentation, there is a description now to do it, when customizing jupyter stacks. I think the mamba command will solve the problem. The commands starting with RUN can also just executed on a linux system.

# Choose your desired base image
FROM jupyter/minimal-notebook:latest

# name your environment and choose the python version
ARG conda_env=python37
ARG py_ver=3.7

# you can add additional libraries you want mamba to install by listing them below the first line and ending with "&& \"
RUN mamba create --quiet --yes -p "${CONDA_DIR}/envs/${conda_env}" python=${py_ver} ipython ipykernel && \
    mamba clean --all -f -y

# alternatively, you can comment out the lines above and uncomment those below
# if you'd prefer to use a YAML file present in the docker build context

# COPY --chown=${NB_UID}:${NB_GID} environment.yml "/home/${NB_USER}/tmp/"
# RUN cd "/home/${NB_USER}/tmp/" && \
#     mamba env create -p "${CONDA_DIR}/envs/${conda_env}" -f environment.yml && \
#     mamba clean --all -f -y

# create Python kernel and link it to jupyter
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --user --name="${conda_env}" && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

# any additional pip installs can be added by uncommenting the following line
# RUN "${CONDA_DIR}/envs/${conda_env}/bin/pip" install --quiet --no-cache-dir

# if you want this environment to be the default one, uncomment the following line:
# RUN echo "conda activate ${conda_env}" >> "${HOME}/.bashrc"

Hope this helps