Alternative for the DYLD_LIBRARY_PATH-trick since Mac OS 10.11 El Capitan with System Integrity Protection

Question

Here is what I have:

• Mac OS 10.11 El Capitan
• python 2.7.12, installed from python.org under /Library/Frameworks/Python.framework/
• PyCharm 2016.2.3
• vtk 7.1.0

Here is what I do:

• Build a python module locally. In my case, this is vtk. For a summary, see the CMake call with which I configure vtk.

  cmake -G Ninja .. -DCMAKE_BUILD_TYPE=Release -DVTK_WRAP_PYTHON=ON -DBUILD_EXAMPLES=OFF -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=OFF -DCMAKE_INSTALL_PREFIX="/opt/dev/versions/vtk/vtk-7.1.0-shared" -DPYTHON_INCLUDE_DIR="/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/" -DPYTHON_LIBRARY="/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib"
  

• Install the python package in a location where python can find it. In my case, this is /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages Note that I am required to extend the DYLD_LIBRARY_PATH by the location where the libs reside: /opt/dev/versions/vtk/vtk-7.1.0-shared/lib/.

• If I start python from the terminal, I can import vtk successfully.

  import vtk
  v = vtk.vtkVersion()
  print v.GetVTKVersion()
  

• If I try to import vtk in PyCharm's python console, I get the following error:

  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "<ipython-input-2-b7e11aadda62>", line 1, in <module>
      import vtk
    File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
      module = self._system_import(name, *args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/vtk/__init__.py", line 41, in <module>
      from .vtkCommonCore import *
    File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
      module = self._system_import(name, *args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/vtk/vtkCommonCore.py", line 9, in <module>
      from vtkCommonCorePython import *
    File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
      module = self._system_import(name, *args, **kwargs)
  ImportError: No module named vtkCommonCorePython
  

By now, I understand that the problem is caused by the System Integrity Protection (SIP) that has been introduced in El Capitan. One of the effects is that child processes only have restricted access to other resources, and most likely, PyCharm executes python as separate process.

I also understand that python cannot import vtk because it cannot find the dylibs which the python module links to. I can verify this in two ways:

• The DYLD_LIBRARY_PATH is empty. This is because python runs as a child process in PyCharm: os.getenv('DYLD_LIBRARY_PATH') returns None.
• When I copy all the libraries from /opt/dev/versions/vtk/vtk-7.1.0-shared/lib/ to the current working directory, I can import the module

Now the question: Apparently, DYLD_LIBRARY_PATH cannot be used in child-processes and hence should not be used anymore at all since El Capitan. So, how to properly replace this "linkage hack" that worked perfectly well before MacOS 10.11.? Is there a way to still use DYLD_LIBRARY_PATH?

Disabling SIP is not an option. Apparently, it helps to copy the dylibs into the current working directory, but this is not feasible for me. Placing the libs in the site-package location (of vtk) doesn't help however.

I'm pretty sure that many people have been relying on the DYLD_LIBRARY_PATH-hack and now struggle with the consequences of SIP - that's why I thought that the community might benefit from this quite lengthy question.

Answer 1

After a long struggle, I was able to solve the last bit of my problem.

By setting a fixed value for the RPATH Run-Path dependent Libraries of the installed binaries, my linking problems are gone.

There are different possibilities to achieve this. I guess one option is to use install_name_tool. For me, the easiest was to build vtk with appropriate CMake flags. Here my updated call to cmake, where CMAKE_MACOSX_RPATH and CMAKE_INSTALL_RPATH make the difference:

    cmake -G Ninja .. -DCMAKE_BUILD_TYPE=Release \
              -DVTK_WRAP_PYTHON=ON \
              -DBUILD_EXAMPLES=OFF \
              -DBUILD_SHARED_LIBS=ON \
              -DBUILD_TESTING=OFF \
              -DCMAKE_INSTALL_PREFIX="/opt/dev/versions/vtk/vtk-7.1.0-shared" \
              -DCMAKE_MACOSX_RPATH=ON \
              -DCMAKE_INSTALL_RPATH="/opt/dev/versions/vtk/vtk-7.1.0-shared/lib" \
              -DPYTHON_INCLUDE_DIR="/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/" \
              -DPYTHON_LIBRARY="/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib"

Read here more about CMake's rpath handling. Note that otool -L vtkCommonCorePython.so (for an example) will still write @rpath in the output, but the value still is fixed.

@rpath/libvtkCommonCorePython27D-7.1.1.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libvtkWrappingPython27Core-7.1.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
@rpath/libvtksys-7.1.1.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libvtkCommonCore-7.1.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)