fastest way to share data between a C++ and Python program? [closed]

Question

I'm trying to build a algorithmic trading program. The program has execution flow like this:

Server sends data via websocket -> Python program receives it and sends it to C++ program -> C++ program processes the data and sends some data to Python code -> Python code sends packets to Server

The reason I'm not building the whole thing in C++ is because Broker's API only supports Python and I cannot perform the operations I wish to perform fast enough if I switch to Python.

The frequency of data is going to be at least ~50kb (binary and Json) per second. So far I've found the following alternatives:

1. Embed Python in C++ code. This seems great, but I'm not sure if would be able to import whole library and use classes/methods (broker's client) in C++.

2. Communicate through sending packets (latency is the issue here)

3. Put the data received in SQL database and have C++ query it every X ms. (again, latency)

Is there any better way to do this?

Answer 1

If you are using CPython (the most common implementation of python) then you can create a dynamic library that can be used as a python module. There Boost.Python

Which can be used as:

#include <boost/python.hpp>
char const* greet()
{
   return "hello, world";
}

BOOST_PYTHON_MODULE(hello_ext)
{
    using namespace boost::python;
    def("greet", greet);
}

> import hello_ext   
> print(hello_ext.greet())
hello, world

---

To build with python 3.7 and boost 1.68.0 you can use following CMake file

cmake_minimum_required(VERSION 3.9.0 FATAL_ERROR)

project("boost_python_sample" LANGUAGES CXX)

set(BOOST_ROOT "C:/local/boost_1_68_0")
find_package(Boost REQUIRED COMPONENTS python37)
set(Python3_ROOT_DIR "C:/python37")
find_package(Python3 REQUIRED COMPONENTS Development)

add_library("boost_python_sample" SHARED "main.cpp")
target_link_libraries("boost_python_sample" Boost::python37 Python3::Python)
target_compile_definitions("boost_python_sample" PUBLIC "BOOST_PYTHON_STATIC_LIB")

Answer 2

You can implement your C++ code as shared library (so or dll). Your interface should be extern "C". Then you can call your native functions directly in python and pass your data via pointers within the same process and memory. To call the native functions you can use Python CTypes.

Answer 3

One way to exchange data between python and C++ is to use a message queue library. One possible library which is designed to be fast is ØMQ (zeroMQ). Is this the fastest way? It depends on your use case. It might be worth evaluating. Especially considering the easy to implement part, good documentation and community support.