What is the most efficient way of sharing a Julia object with C++?

Question

I have a storage manager written in C++, and I want to pass some objects from Julia to the C++ program. It's enough for me to receive the content as an array of bytes that can later be passed back to Julia and be easily decoded.

What is the best approach that minimizes the number of copying data around (and also avoids writing/reading to/from disk)?

It's possible to allocate the required memory from the C++ program and share it with Julia to serialize the object, or get a pointer to the allocated memory from Julia into the C++ program. In the latter case, I am not sure how to prevent garbage collection from Julia side. Also, I do not know which serialization/deserialization method is more suitable for such a use case.

Would you please guide me to find the best approach for this kind of lightweight serialization/deserialization between Julia and C++?

edit: if the answer is OS-dependent, please give the answer for Linux or macOS.

Answer 1

Unfortunately, I cannot provide you with any source code since I haven't been using any of the frameworks mentioned below in C++ nor do I know Julia at all.

The approach I've been following in a past project isn't as lightweight as sharing pointers to the memory already allocated by C++. However, since there hasn't been an answer yet, I just want to add my two cents on how I've been exchanging objects between programming languages (Java and C# in my case).

Instead of reading from and writing to the disc I've used a messaging queue to make both sub-systems exchange objects with each other.

Serialization

Serialization, especially when it comes to more complex objects can be quite resource-hungry. However, I've come to like using protocol buffers for that matter as they serialize the pre-defined object into a stream of bytes.

An example for a C++ library for protocol buffers can be found in the Google Protobuffers. Respectively, a protocol buffer for Julia can be found in ProtoBuf.jl

The downside about using protocol buffers is that you need to have a defined message format which means that you already need to know what the objects you want to exchange look like beforehand.

Exchanging data

A data exchange could be done using a messaging queue which instead of sharing pointers or writing to the disc uses the network interface of the local machine. A messaging queue I've been using already has been ZeroMQ as it is quite lightweight and fairly easy to use. Any other messaging queues should work for that purpose too though.

A Julia interface for ZeroMQ is ZMQ.jl and cppzmq respectively for C++. The complete guide for ZeroMQ can be really helpful to get you started but you won't need the whole documentation for your purpose.