What is libpython3.so compared with libpython3.5m.so built from python 3.5.2 source?

Question

In my application, I use boost_python and python 3.5.2. All built from source in Ubuntu 14.

When I built Python 3.5.2 from source with --with-shared options in Ubuntu, I got libpython3.so (7.6kB) and libpython3.5m.so (12MB). I assume the big one is the real one and the small one might be something forwarding calls to the real interfaces.

As boost_python might assume the client to link against python (https://svn.boost.org/trac/boost/ticket/2615), I linked libpython3.so with my application. But when I run it, I got the unresolved symbols error.

ldd -r myapp or ldd -r libboost_python.so both listed all python symbols unresolved which could be found in nm -D libpython3.5m.so.

# ldd -r lib/libboost_python3.so
    linux-vdso.so.1 => (0x00007ffe767fb000)
    libstdc+.so.6 => /usr/lib/x86_64-linux-gnu/libstdc+.so.6 (0x00007f130a7a3000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f130a58d000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f130a1c8000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1309ec2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f130acf4000)
undefined symbol: PyExc_ImportError (lib/libboost_python3.so)
undefined symbol: PyProperty_Type (lib/libboost_python3.so)
undefined symbol: PyExc_StopIteration (lib/libboost_python3.so)
undefined symbol: PyBool_Type (lib/libboost_python3.so)
undefined symbol: PyExc_ValueError (lib/libboost_python3.so)
undefined symbol: PyList_Type (lib/libboost_python3.so)
undefined symbol: _Py_NotImplementedStruct (lib/libboost_python3.so)
undefined symbol: PyExc_TypeError (lib/libboost_python3.so)
undefined symbol: PyDict_Type (lib/libboost_python3.so)
...

libpython3.so depend on libpython3.5m.so but itself has no those symbols.

I think based on that I should link my application with libpython3.5m.so instead of with libpython3.so. But the weird thing is that if I use LD_PRELOAD to load the libpython3.so, those symbols are found in ldd -r libboost_python3.so

# LD_LIBRARY_PATH=lib LD_PRELOAD=lib/libpython3.so ldd -r lib/libboost_python3.so
    linux-vdso.so.1 => (0x00007ffcb51f0000)
    lib/libpython3.so (0x00007f6f728e3000)
    libstdc+.so.6 => /usr/lib/x86_64-linux-gnu/libstdc+.so.6     (0x00007f6f725df000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f6f723c9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6f72004000)
    libpython3.5m.so.1.0 => lib/libpython3.5m.so.1.0 (0x00007f6f71ae1000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0     (0x00007f6f718c3000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6f715bd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6f72d32000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6f713b9000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f6f711b6000)

Why having the libpython3.so and how to use it? Or shall I just only use the libpython3.5m.so directly?

Answer 1

The libpython3.so library is there to support PEP 384 -- Defining a Stable ABI.

Historically, Python has not guaranteed ABI stability at the C level between minor version releases (e.g. between 3.4 and 3.5). They might be source compatible but certain structures might change size, or structure members change type, etc. However, there are portions of the ABI that are mature and have remained stable over a longer period.

The Stable ABI PEP identified a subset of the Python C API that was stable wouldn't place undue restrictions on future Python development if the developers committed to maintaining binary compatibility for the subset. If a program or extension limited itself to only using this subset, then it could theoretically be used across different Python versions without recompilation.

Once you've compiled some code using the stable ABI, there is still the question of how to link to the runtime. For Python 3.5.x, you'd want to link using -lpython3.5m. For Python 3.6.x, you want -lpython3.6m. This is where libpython3.so comes in.

The libpython3.so library only has one purpose: for Python 3.5 it links to libpython3.5m.so, and on 3.6 it links to libpython3.6m.so, etc. So if an extension links using -lpython3, it will have access to the runtime of the version of Python installed on the system.

Now back to your original problem: unless you are absolutely sure that you are only using features found in the stable ABI (which in your case means finding out whether libboost_python only uses the stable ABI), then you probably want to link to the versioned libpython3.5m.so.

If in doubt, you're better off linking to the versioned library: it is much easier to debug a dynamic link error than a segfault due to ABI changes, should you upgrade to a newer version of Python.