Compiling C shared library with distutils' setup.py, when the library depends on a second shared library

Question

I'm on OSX, trying to compile a shared library in C with distutils' setup.py (to use in python using ctypes). I'm new to distutils, but I'm having problems when the shared library I want to compile (libreboundx.so) depends on another shared library (librebound.so). Explicitly, in modify_orbits_direct.c I have

#include "rebound.h"

rebound.h is in directory /Users/dt/rebound/src/, and all the functions in rebound.h are in the shared library librebound.so, which is in /Users/dt/rebound/.

The linking with cc would look like.

cc -fPIC -shared reboundx.o -L/Users/dt/rebound -lrebound -o libreboundx.so

UPDATE: This situation looks exactly like the example at the end of Sec. 3 at https://docs.python.org/2/extending/building.html. I've updated my setup.py to mimic that one:

libreboundxmodule = Extension('libreboundx',
                sources = [ 'src/reboundx.c',
                            'src/modify_orbits_direct.c'],  
                include_dirs = ['src', '/Users/dt/rebound/src'], 
                extra_compile_args=['-fstrict-aliasing', '-O3','-std=c99','-march=native', '-D_GNU_SOURCE', '-fPIC'],
                library_dirs=['/Users/dt/rebound'],
                libraries=['rebound'],
                                )   

This installs fine when I run

pip install -e ./

Build output:

You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Obtaining file:///Users/dtamayo/Documents/workspace/reboundx
Installing collected packages: reboundx
Running setup.py develop for reboundx
Successfully installed reboundx-1.0

but when I try

import reboundx

in Python, I get an OSError: dlopen(libreboundx.so, 10): Symbol not found: _reb_boundary_particle_is_in_box, which is a function in the other library (librebound.so), which doesn't even get called in the code for libreboundx.so.

If I link the shared library with the cc command above, everything works, and I can use the shared library libreboundx.so perfectly fine in C. If I try to take the same libreboundx.so I compile with the cc command and stick it where setup.py would put it, then try to import reboundx in python, I instead get

OSError: dlopen(/Users/dtamayo/Documents/workspace/reboundx/reboundx/../libreboundx.so, 10): Library not loaded: librebound.so

Referenced from: /Users/dtamayo/Documents/workspace/reboundx/libreboundx.so Reason: image not found

Could this be like an rpath issue, where at runtime libreboundx.so doesn't know where to look for librebound.so?

Answer 1

Thanks for all the suggestions. I should have specified in the question that in the end I want a solution that I could package for upload to PyPy so users can install with a single command. It should also run on both OSX and Linux, so I preferred solutions not involving install_name_tool.

I haven't been able to test it, but I think adding

 runtime_library_dirs=['/Users/dt/rebound'],

next to library_dirs should fix the problem on Linux. Apparently this doesn't work on Mac, but you instead can use extra_link_args. Adding this below the libreboundxmodule definition posted above,

if platform.system() == 'Darwin':
    extra_link_args.append('-Wl,-rpath,'+'/Users/dtamayo/Documents/workspace/rebound')

fixed my problem. I found the answer here: Python runtime_library_dirs doesn't work on Mac