How to compile a whole Python library (including dependencies) so that it can be used in C?

Question

How to compile a whole Python library along with it's dependencies so that it can be used in C (without invoking Python's runtime). That is, the compiled code has the Python interpreter embedded and Python does not need to be installed on the system.

From my understanding, when Python code is compiled using Cython it:

• Does not invoke the python runtime if the --embed argument is used
• Compiles files individually
• Allows for different modules to be called (from the Python runtime / other compiled Cython files)

The question which are still unclear are:

• How to use these module files from C? Can the compiled Python files call other compiled Python files when used in C?
• Does only the library entry point need to be declared or do all functions need to be declared?
• How to manage the Python dependencies? how to compile them too (so that the Python runtime is not needed).

A simplified example for a python library called module where __init__.py is an empty file:

module/
├── run.py
├── http/
│   ├── __init__.py
│   ├── http_request.py

http_requests.py contains:

import requests

def get_ip():
    r = requests.get('https://ipinfo.io/ip')
    print(r.text)

and run.py contains the following:

from http import http_request

if __name__ == '__main__':
    http_request.get_ip()

How to call the function get_ip from C without using the Python runtime (needing to have Python installed when running the application).

The above example is very simple. The actual use case is collecting/processing robotics data in C at a high sampling rate. Whilst C is great for basic data processing there are excellent Python libraries which allow for much more comprehensive analysis. The objective would be to call the Python libraries on data which has been partially processed in C. This would allow us to get a much more detailed understanding of the data (and process it in "real time"). The data frameworks are way too large for our team to rewrite in C.

Answer 1

How to compile a whole Python library along with it's dependencies so that it can be used in C (without invoking Python's runtime).

This is impossible in general. Python code is practically expected to run on a Python interpreter.

Sometimes, when only a small subset of Python is used (even indirectly by everything your Python code is using) you might use Cython (which is actually a superset of a small subset of Python: a lot of genuine Python features cannot be used from Cython, or uses the Python interpreter). But not every Python code can be cythonized, since Python and C have a very different (and incompatible) semantics (and memory management).

Otherwise (and most often), the C code using your Python stuff should embed the Python interpreter.

A wiser and more robust approach, if your goal is to make a self-sufficient C library usable from many C programs (on systems without Python), is to rewrite your code in C.

You could also consider starting (in your C library) some Python process (server-like, doing your Python stuff) and using inter-process communication facilities, that would be operating system specific. Of course Python needs to be installed on the system of the application using your library. For example, for Linux, you might fork some Python process in your library, and use pipe(7) or unix(7) sockets to communication from the C library to that process (perhaps using something like JSONRPC).

Your edit (still not an MCVE) shows some HTTP interaction done in Python. You could consider doing that in C, with the help of HTTP client libraries in C like libcurl, or (if so needed) of HTTP server libraries like libonion.

So consider rewriting your stuff in C but using several existing C libraries (how and what to choose is a very different question, probably off-topic on StackOverflow). Otherwise, accept the dependencies on Python.

The actual use case is collecting/processing robotics data in C at a high sampling rate. Whilst C is great for basic data processing there are excellent Python libraries which allow for much more comprehensive analysis.

You could keep high-level things in Python (see this) but recode low level things in C to accelerate them (many software are doing that, e.g. TensorFlow, ...), perhaps as extensions in C for Python or in some other process. Of course, that means some development efforts. I don't think that avoiding Python entirely is reasonable (getting rid of Python entirely is not pragmatical), if you use a lot of code in Python. BTW, you might perhaps consider embedding some other language in your C application (e.g. Lua, Guile, Ocaml - all of them are rumored to be faster than Python) and keep Python for the higher level, running in some other process.

You need to put more efforts on the architectural design of your thing. I'm not sure that avoiding Python entirely is a wise thing to do. Mixing Python and C (perhaps by having several processes cooperating) could be wiser. Of course you'll have operating system specific stuff (notably on the C side, for inter process communication). If on Linux, read something about Linux system programming in C, e.g. ALP or something newer.