How to I transfer an image(opencv Matrix/numpy array) from c++ publisher to python sender via ZeroMQ?

Question

I know how to send a string message from c++ to python via zeromq.

Here's the code for sending a string message I know :

C++ sender code :

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://localhost:5563");
std::string message = "Hello from sender";
const char *message_char = message.c_str();
zmq_send(publisher, message_char, strlen(message_char), ZMQ_NOBLOCK);

Python receiver code :

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://*:5563")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
message = receiver.recv_string()

What I want is to send an image from c++ zeromq publisher to python receiver.

Answer 1

Disclaimer : Answering my own question, so that others shall not stuck where I did.

So lets get started.

What is Zero MQ ?

ZeroMQ is a high-performance asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.

Before, we get started here are the basics :

Protocol/Library used : ZeroMQ

Publisher : C++ oriented

Subsciber : Python oriented

---

Sending String/char array message via ZeroMQ :

C++ Publisher :-

// Setting up ZMQ context & socket variables
void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB); 
int bind = zmq_bind(publisher, "tcp://*:9000");
std::string message = "Hello from sender";
const char *message_char = message.c_str(); // Converting c++ string to char array
// Sending char array via ZMQ
zmq_send(publisher, message_char, strlen(message_char), ZMQ_NOBLOCK);

Python Subscriber :-

// Setting up ZMQ context & socket variables
context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
// Subscribing to start receiving messages
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
message = receiver.recv_string()

---

Sending Image/ndarray array message via ZeroMQ :

For handling images, opencv is an awesome library. Simple, easy to code & lightning fast.

C++ Publisher :-

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://*:9000");

// Reading the image through opencv package
cv::Mat image = cv::imread("C:/Users/rohit/Desktop/sample.bmp", CV_LOAD_IMAGE_GRAYSCALE );
int height = image.rows;
int width = image.cols;
zmq_send(publisher, image.data, (height*width*sizeof(UINT8)), ZMQ_NOBLOCK);

In above code, image is read as a grayscale image, you can read 3-channel (RGB) image as well, by passing appropriate parameters in opencv's imread method.

Also remember to modify the size (3rd parameter in zmq_send function call) accordingly.

Python Subscriber :-

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
// Receiving image in bytes
image_bytes = receiver.recv()
int width = 4096; // My image width
int height = 4096; // My image height
// Converting bytes data to ndarray
image = numpy.frombuffer(image_byte, dtype=uint8).reshape((width, height))

TO DO / IMPROVEMENT : You can also pass the image size from the c++ publisher along with the image data. So, that image can be reshaped accordingly at python side.

ZMQ_SNDMORE flag comes handy here

Just add another zmq_send statement at c++ side.

zmq_send(publisher, img_height, strlen(img_height), ZMQ_SNDMORE)
zmq_send(publisher, img_width, strlen(img_width), ZMQ_SNDMORE)
zmq_send(publisher, image.data, (height*width*sizeof(UINT8)), ZMQ_NOBLOCK);

Similarly, add corresponding receiving statements at python end.

height = receiver.recv_string(ZMQ_RCVMORE)
width = receiver.recv_string(ZMQ_RCVMORE)
image_bytes = receiver.recv()

---

Another Improvement

Thanks @Mark Setchell for pointing out an improvement.

Sending opencv Matrix of large size directly over network can be costly. A better approach would be to encode the image before sending over the network.

C++ Publisher :-

void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int bind = zmq_bind(publisher, "tcp://*:9000");

// Reading the image through opencv package
cv::Mat image = cv::imread("C:/Users/rohit/Desktop/sample.bmp", CV_LOAD_IMAGE_GRAYSCALE );
int height = image.rows;
int width = image.cols;
cv::vector<uchar> buffer;
cv::imencode(".jpg", image, buffer);
zmq_send(publisher, buffer.data(), buffer.size(), ZMQ_NOBLOCK);

Python Subscriber :-

context = zmq.Context()
receiver = context.socket(zmq.SUB)
receiver.connect("tcp://localhost:9000")
receiver.setsockopt_string(zmq.SUBSCRIBE, "")
// Receiving image in bytes
image_bytes = receiver.recv()
// Decoding the image -- Python's PIL.Image library is used for decoding
image = numpy.array(Image.open(io.BytesIO(image_byte)))