How to keep `LD_LIBRARY_PATH` from getting overwritten to pwd when running python script

Question

I have a python script which eventually calls a binary that requires certain shared libraries. However, I have been running into the following error:

error while loading shared libraries: libmkl_rt.so: cannot open shared object file: No such file or directory

I have narrowed down the cause of this error to the fact that python overwrites LD_LIBRARY_PATH to be the PWD. When executing the binary manually via the bash, everything works properly as this environment variable is properly set. Here are examples of the current behavior:

$ echo $LD_LIBRARY_PATH
/the/proper/paths

$ python
>>> import os
>>> os.environ.get('LD_LIBRARY_PATH')
/the/proper/paths

$ python script.py
/my/present/working/directory

Here, script.py has the same contents as the second example.

Other information asked for:

$ which python
~/.pyenv/shims/python
$ python --version
Python 3.7.3
$ type python
python is hashed (/home/ec2-user/.pyenv/shims/python)
$ realpath `which python`
/home/ec2-user/.pyenv/shims/python

I would like to know if there's a way to see where this variable is getting changed and/or why this would be happening.

I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

Answer 1

I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

What's happening under the hood is that some code in your script.py (or code which script.py imports) sets the environment variable.

You can run gdb --args python script.py, set a breakpoint on setenv and putenv, and have GDB stop the program when the environment is reset. However, this will stop somewhere in the Python runtime, and figuring out which Python code is running might get tricky.

---

That said, setting LD_LIBRARY_PATH is not the right way to fix the "binary that requires certain shared libraries" problem in the first place.

A much better solution is to either link the binary with correct -rpath flag, or to fix already-linked binary using patchelf --set-rpath, so the binary runs correctly without LD_LIBRARY_PATH.