Data transfer between C++ and Python

Question

I would like to share memory between C++ and Python.

My problem:

1. I am working with big data sets (up to 6 GB of RAM) in C++. All calculations are done in c++.
2. Then, I want to "paste" all my results to a Python program. But I can only write my data on disk, and then read that file from Python, which is not efficient.

Is there any way to "map" memory corresponding to C++ variables so that I may access the data from Python? I don't want to copy 6GB of data onto a hard drive.

Answer 1

First path: I think the more appropriate way for you to go is ctypes. You can create a shared library, and then load the functions of the shared library in Python, and fill all the data containers you want in Python.

In Windows, you can create a DLL, and in Linux you can create a shared .so library.

Now this has the advantage that this will be independent of your Python version.

Second path: I think it's less appropriate but you can get it to work, which is the Python C Extension. With this, you can call Python data containers (PyObjects) and fill them inside C.

However, the code you compile here will always need to be linked to Python libraries.

Which one to use?:

• Use ctypes if you have some functions you want to call in C/C++, and then do the rest of the work in Python.
• Use Python C Extension if you have some functions you want to call in Python, and you want to do the rest in C/C++.

With both options, you can transfer huge blocks of memory between C++ and Python without necessarily involving any disk read/write operations.

Hope this helps.