VS Code Jupyter integration does not consider custom LD_LIBRARY_PATH

Question

I recently setup a fresh EC2 instance for development running Amazon Linux 2. To run the recent version of prefect (https://orion-docs.prefect.io/) I had to install an up to date version of SQLite3, which I compiled from source. I then set the LD_LIBRARY_PATH environmental variable to "/usr/local/lib", and installed python 3.10.5 with the LDFLAGS and CPPFLAGS compiler arguments to include that folder as well, so that the new sqlite libraries are found by python. All good so far, when running the jupyter notebook server or the prefect orion server from the terminal everything works fine. If I want to use the integrated jupyter environment from VS Code I run into the issue that the kernel does not start:

Failed to start the Kernel.

ImportError: /home/mickelj/.pyenv/versions/3.10.5/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so: undefined symbol: sqlite3_trace_v2.

This leads me to believe that the system sqlite library is used, as this is the same error I get when I unset the LD_LIBRARY_PATH env variable. However when calling ldd /home/mickelj/.pyenv/versions/3.10.5/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so I am getting the following:

linux-vdso.so.1 (0x00007ffcde9c8000)
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x00007f96a3339000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f96a311b000)
libc.so.6 => /lib64/libc.so.6 (0x00007f96a2d6e000)
libz.so.1 => /lib64/libz.so.1 (0x00007f96a2b59000)
libm.so.6 => /lib64/libm.so.6 (0x00007f96a2819000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f96a2615000)
/lib64/ld-linux-x86-64.so.2 (0x00007f96a3870000)

Where the new sqlite3 library is correctly referenced. If I unset the LD_LIBRARY_PATH variable the second line changes to:

libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f9dce52e000)

So my guess is that the VS Code jupyter integration does not consider environment variables, so my question is: is there a way to specify them (and in particular the LD_LIBRARY_PATH) globally for VS Code or for the built-in jupyter server at runtime or anywhere else to fix this?

Answer 1

Using ipykernel to create a custom kernel spec with env variable solved this for me.

Steps:

1. Create a kernelspec with your environment.

conda activate myenv # checkout that venv, using conda as an example

# pip install ipykernel # in case you don't have one

python -m ipykernel install --user --name myenv_ldconf

2. Edit the kernelspec file, add env variable in the object

nano ~/.local/share/jupyter/kernels/myenv_ldconf/kernel.json

You will see something like this:

{
 "argv": [
  "/home/alice/miniconda3/envs/myenv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "myenv_ldconf",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

After adding env variable:

{
 "argv": [
  "/home/alice/miniconda3/envs/myenv/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "myenv_ldconf",
 "language": "python",
 "env": {"LD_LIBRARY_PATH": "/home/alice/miniconda3/envs/myenv/lib"},
 "metadata": {
  "debugger": true
 }
}

Ref: How to set env variable in Jupyter notebook

3. Change the kernel in vscode to myenv_ldconf.