How to get the path of the Python library from within Python

Question

Initial situation:

I have a C/C++ library and provide bindings for different target languages. One of these languages is Python. I decided to use Swig because it provides good support for a lot of languages and its integration into CMake is straightforward.

In order to improve the usability of the Python binding and it's installation I would like to provide a setup.py. The idea is to run CMake from within the setup.py file because CMake contains all the logic of how to create the binding correctly.

What I currently got:

I'm able to run CMake and build the target of the Python binding from within setup.py.

By design of Swig the CMake file needs to distinguish between Python 2 and Python 3. While running CMake it detects the location of the Python installation and configures the environment. If the user installed both, Python 2 and Python 3 (with their development packages) CMake always takes Python 3.

According to: https://cmake.org/cmake/help/v3.0/module/FindPythonLibs.html the actual used Python version can be specified by setting PYTHON_LIBRARY and PYTHON_INCLUDE_DIR. For example I have to run:

cmake -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so

What is the problem:

If the user executes:

python setup.py build

the Python version of the executable 'python' might be 2 but CMake builds the binding using version 3 (see above).

According to Find python header path from within python? I'm able to get the location of the header file by using:

from distutils.sysconfig import get_python_inc
get_python_inc()

Unfortunately, I don't know how to get the path of the Python library. The solution provided by: Distribution independent libpython path doesn't work for me because it always returns '/usr/lib' instead of '/usr/lib/x86_64-linux-gnu/libpython2.7.so'

The Question:

How do I get the location (the full path) of the Python library from within Python.

Answer 1

The Question:

How do I get the location (the full path) of the Python library from within Python.

The Answer:

Looking for INSTALL_SHARED in the Makefile for python in: https://github.com/python/cpython/blob/3.5/Makefile.pre.in

We can find:

$(INSTALL_SHARED) libpython$(VERSION)$(SO) $(DESTDIR)$(LIBDIR)/$(INSTSONAME); \

We can see there is no single variable with the asked information. You need to build it from two others:

from distutils.sysconfig import get_config_var
libpython = "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))

Testing in python 2.7:

Python 2.7.13 (default, Nov 20 2017, 02:33:01)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))
'/usr/lib64/libpython2.7.so.1.0'

Testing in python 3.3:

Python 3.3.0 (default, Nov 11 2013, 09:56:47)
[GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils.sysconfig import get_config_var
>>> "%s/%s" % (get_config_var("LIBDIR"), get_config_var("INSTSONAME"))
'/usr/lib64/libpython3.3m.so.1.0'

So, it just works for CPython (>=2.3) in linux as you can see in the python Makefile and there is no guarantee that it will work on the future, anyway, this hasn't changed in the last 14 years, since 11th May 2003.

You may use the BINDIR and DLLLIBRARY variables for Windows instead, as you can see from the Makefile:

$(INSTALL_SHARED) libpython$(VERSION)$(SO) $(DESTDIR)$(BINDIR); \

However, I have no means to test in Windows.

Answer 2

I don't know Python a lot and I didn't manage to find a good answer to that problem of figuring out the full path and name of the Python's library. So I tried to come up with a solution by making tests using:

• CentOS 7 (Python 3.6)
• Ubuntu 19.10 (Python 3.7)
• Windows 10 (Python 3.8 from Chocolatey)

I used CMake's find_package(Python3) as a reference, and these are the libraries it found on each of those platforms:

• CentOS: /usr/lib64/libpython3.6m.so
• Ubuntu: /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.so
• Windows: C:/Python38/libs/python38.lib

From that, I tried to find out if there was any combination of configuration variables (as reported by distutils.sysconfig.get_config_vars()) that would give me those paths and library names CMake found. It seemed to me that unfortunately there is no single way that works portably across all these platforms to get the full library path. What I could get of it is that these combinations seem to work on each platform:

• CentOS: LIBDIR + LDLIBRARY
• Ubuntu: LIBPL + LDLIBRARY
• Windows: BINDIR + hardcoding libs\python3.lib