How to send OpenCV video footage over ZeroMQ sockets?

Question

I've got a simple webcam which I read out using OpenCV and I'm now trying to send this video footage to a different (Python) program using ZeroMQ. So I've got the following simple script to read out the webcam and send it using a ZeroMQ socket:

import cv2
import os
import zmq
import base64

context = zmq.Context()
footage_socket = context.socket(zmq.PUB)
footage_socket.connect('tcp://localhost:5555')

# init the camera
camera = cv2.VideoCapture(0)

while True:
    try:
        (grabbed, frame) = camera.read()            # grab the current frame
        frame = cv2.resize(frame, (640, 480))       # resize the frame
        footage_socket.send_string(base64.b64encode(frame))

        # Show the video in a window
        cv2.imshow("Frame", frame)                  # show the frame to our screen
        cv2.waitKey(1)                              # Display it at least one ms
        #                                           # before going to the next frame

    except KeyboardInterrupt:
        camera.release()
        cv2.destroyAllWindows()
        print "\n\nBye bye\n"
        break

This works well in that it shows the video and doesn't give any errors.

I commented out the two lines which show the image (cv2.imshow() and cv2.waitKey(1)). I then started the script below in paralel. This second script should receive the video footage and show it.

import cv2
import zmq
import base64
import numpy as np

context = zmq.Context()
footage_socket = context.socket(zmq.SUB)
footage_socket.bind('tcp://*:5555')
footage_socket.setsockopt_string(zmq.SUBSCRIBE, unicode(''))

# camera = cv2.VideoCapture("output.avi")

while True:
    try:
        frame = footage_socket.recv_string()
        frame = np.fromstring(base64.b64decode(frame), dtype=np.uint8)
        cv2.imshow("Frame", frame)                  # show the frame to our screen
        cv2.waitKey(1)                              # Display it at least one ms
        #                                           # before going to the next frame
    except KeyboardInterrupt:
        cv2.destroyAllWindows()
        break

print "\n\nBye bye\n"

Unfortunately, this freezes on cv2.waitKey(1).

Does anybody know what I'm doing wrong here? Do I need to decode the footage differently? All tips are welcome!

Answer 1

In the end I solved the problem by taking intermediate steps. I first wrote individual images to disk, and then I read out those images again. That led me to the fact that I needed to encode the frame as an image (I opted for jpg), and with the magic methods cv2.imencode('.jpg', frame) and cv2.imdecode(npimg, 1) I could make it work. I pasted the full working code below.

This first script reads out the webcam and sends the footage over a zeromq socket:

import cv2
import zmq
import base64

context = zmq.Context()
footage_socket = context.socket(zmq.PUB)
footage_socket.connect('tcp://localhost:5555')

camera = cv2.VideoCapture(0)  # init the camera

while True:
    try:
        (grabbed, frame) = camera.read()  # grab the current frame
        frame = cv2.resize(frame, (640, 480))  # resize the frame
        encoded, buffer = cv2.imencode('.jpg', frame)
        footage_socket.send_string(base64.b64encode(buffer))
        
    except KeyboardInterrupt:
        camera.release()
        cv2.destroyAllWindows()
        print "\n\nBye bye\n"
        break

and this second script receives the frame images and displays them:

import cv2
import zmq
import base64
import numpy as np

context = zmq.Context()
footage_socket = context.socket(zmq.SUB)
footage_socket.bind('tcp://*:5555')
footage_socket.setsockopt_string(zmq.SUBSCRIBE, unicode(''))

while True:
    try:
        frame = footage_socket.recv_string()
        img = base64.b64decode(frame)
        npimg = np.fromstring(img, dtype=np.uint8)
        source = cv2.imdecode(npimg, 1)
        cv2.imshow("image", source)
        cv2.waitKey(1)

    except KeyboardInterrupt:
        cv2.destroyAllWindows()
        print "\n\nBye bye\n"
        break

Answer 2

Step 0: Inventory of Risks

Given the target is clear, your rapid-prototyping of the distributed application infrastructure depends on several risky points.

0) OpenCV cv2 module heavily uses underlying C-based components and python beauty hangs quite often if trying to use cv2.imshow() facilities and cv2 ( external FSA ) window-management & frame-display services.

1) ZeroMQ framework can help you a lot more than to try to enforce data to be cast into (string) just to use .send_string() / .recv_string() - may get better here once rather moving the known image ( pxW * pxH )geometry * RGB-depth inside some smarter BLOB-mapped object ( will speak about this aspect a bit more in architecture outlook below ).

2) Given points 0 & 1, the ZeroMQ infrastructure ( the more in prototyping ) ought become robust against unhandled exceptions ( which would leave zmq.Context() instances & their associated .Socket() children hanging on their own on allocated resources in case cv2.imshow() crashes, as it quite often does so in rapid prototyping loops.

Thus a thoroughful & self-disciplined framing of the code inside a try: except: finally: handler & explicit initial .setsockopt( zmq.LINGER, 0 ) immediately upon socket instantiation + final .close() + context.term() inside the finally: handler section are a fair must.

---

Step 1: Validate the ZeroMQ part by sending a SEQ of plain int-s

The best first level of problem isolation is to setup the flow, just to deliver an uncontrolled SEQ of integers, being .send( ) broadcast from PUB side.

...                                                   # FOR STREAMING, ALWAYS PREFER DESIGNS USING A NONBLOCKING MODE
SEQ += 1
footage_socket.send( SEQ, zmq.NOBLOCK )               # PUB.send( SEQ ) -> *SUB*
...

Unless your receiving side demonstrates it's robust handling of a flow of int-s, it makes no sense to proceed any further.

...
aMsgIN = footage_socket.recv( zmq.NOBLOCK )           # FOR STREAMING, ALWAYS PREFER DESIGNS USING A NONBLOCKING MODE
print "{0:}".format( aMsgIN if len( aMsgIN ) > 0 else "." ),
# sleep(...)                                          # backthrottle the loop a bit
...

---

Step 2: Decouple .imshow() from .recv() data & event-loops

In case your data-pumps work as needed, the remote display starts to make sense as a next target.

ZeroMQ delivers either a complete message ( a BLOB ), or nothing. This is a fact. Next, ZeroMQ does not warrant anyone to deliver error-free. These are the facts, your design has to live with.

The acquisition is the simpler part, just grab the data ( maybe some colourspace conversions may take centrally here, but otherwise, the task is ( for sub 4K / sub 30fps image processing ) typically error-free on this side.

frame, as acquired above, is a numpy.ndarray instance. The best performance would be received on sending a hard-coded, binary-mapped BLOB, without any "smart" conversions, as obviously, the frame is just a big pack of bits ( though there are some more advanced delicacies possible with Zero-copy mechanics of ZeroMQ, but these would make no direct benefit in this stage on the sender side ).

#                               struct.unpack() / .pack() seems "just"-enough for fast & smart custom Payload protocol designs
DecodeWireMSG_DATA( aMSG_DATA = struct.unpack( "!" + ( aPayloadHEADER + ( ( pxW * pxH ) * RGB_DEPTH * CELL_SIZE ) ) * "I", aMsgIN ) )

The harder part is on the receiver side. If one tries to use the cv2 built-in event-loop engine, hidden inside .imshow(), this loop will collide with your external logic of reading a flow of updates via .recv() as published from the PUB side.

As a reasonable compromise, one may ignore all the "delayed" frame-s, that did not get processes in sync with the PUB-side acquisition / broadcast cadency, and just display the most recent one ... using zmq.CONFLATE option on the ZeroMQ transport infrastructure ( delayed "old" pictures have lost their sense if the purpose of the reconstruction of the flow-of-events is just a visual preception, on the contrary, if the purpose is to document the complete acquisition 1:1, the zmq.CONFLATE would be discarding frame instances, which ought get processed, so some other architecture ought be added for such a 1:1 documentographic purpose, best separate from the "just-visual" branch of the flow of data / processing ).

Having done this, the .recv() ( might be a composition of a Poller() + .recv() loop ) will provide the SUB side an appropriate data-pump, that is independent from the cv2 tools and it's .imshow() ( hidden )-FSA event-loop.

---

Architecture & Performance Hints:

• for higher fps + FullHD / 2K / 4K projects, profile systematically the accumulated processing latencies / delays of the cv2 processing, using zmq.Stopwatch() instances' { .start(), .stop() } methods.

• having hard data, you may in time detect, when additional needs appear to solve some even harder-real-time constraints, so think about:

• principally avoid any risk of falling into any uncontrollable black-hole of python garbage collection -- always control { gc.disable() | gc.enable();gc.collect() } surrounding your critical-path sections and launching an explicit gc.collect() where your design knows it is feasible.

• avoid new memory allocation delays -- may pre-allocate all necessary numpy arrays and later just enforce numpy to use for those an inplace mode of data modifications, thus avoiding any further ad-hoc memory management associated wait-states

• design the traffic for increased error-immunity with separate, multi-streamed, independent updates of just sections ( stripes ) of the whole large / (colour)-deep image ( remember Zero-Warranty - getting either a complete "fat"-message or None )

• tune-up the .Context()'s performance using zmq.AFFINITY to map different classes of I/O traffic priorities onto segregated zmq.Context( N ) I/O-threads.

• fine-tune zmq.SNDBUF + zmq.SNDHWM on PUB side, if multi-subscribers are expected and zmq.CONFLATE is not used.

• last but not least, may harness numba.jit() LLVM-pre-compiled acceleration of re-useable code for the critical-path functions ( typically the heavy numpy processing ), where additional microseconds shaved-off bring their most beneficial effects on your video-processing pipeline, while still remaining in the comfort of pure python ( well, sure, still with some cv2 caveats ).

---

Some more hints & tricks with cv2 in prototyping phase:

May like this for cv2 based image-processing.

May like this for cv2 simple GUI-interactive parameter tuning of methods.

May like this for cv2 processing pipeline profiling with zmq.Stopwatch() details down to [usec]