Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Multithreaded cv2.imshow() in Python does not work

I have two cameras (using OpenNI, I have two streams per camera, handled by the same instance of the driver API) and would like to have two threads, each capturing data from each camera independently, i.e. for one instance of the driver API, say cam_handler, I have two streams depth and rgb per camera, say cam_handler.RGB1_stream and cam_handler.DEPTH1_stream

Here is the code for the same:

import threading

def capture_and_save(cam_handle, cam_id, dir_to_write, log_writer, rgb_stream,
                     depth_stream, io):
    t = threading.currentThread()
    shot_idx = 0
    rgb_window = 'RGB' + str(cam_id)
    depth_window = 'DEPTH' + str(cam_id)
    while getattr(t, "do_run", True):
        if rgb_stream is not None:

            rgb_array = cam_handle.get_rgb(rgb_stream)
            rgb_array_disp = cv2.cvtColor(rgb_array, cv2.COLOR_BGR2RGB)
            cv2.imshow(rgb_window, rgb_array_disp)
            cam_handle.save_frame('rgb', rgb_array, shot_idx, dir_to_write + str(cam_id + 1))
            io.write_log(log_writer[cam_id], shot_idx, None)

        if depth_stream is not None:
            depth_array = cam_handle.get_depth(depth_stream)
            depth_array_disp = ((depth_array / 10000.) * 255).astype(np.uint8)
            cv2.imshow(depth_window, np.uint8(depth_array_disp))
            cam_handle.save_frame('depth', depth_array, shot_idx, dir_to_write + str(cam_id + 1))
        shot_idx = shot_idx + 1

        key = cv2.waitKey(1)
        if key == 27:  # exit on ESC
            break
    print "Stopping camera %d thread..." % (cam_id + 1)
    return

def main():
    # Setup camera threads
    cam_threads = []
    dir_to_write = "some/save/path"
    for cam in range(cam_count):
        cam = (cam + 1) % cam_count
        cv2.namedWindow('RGB' + str(cam))
        cv2.namedWindow('DEPTH' + str(cam))
        one_thread = threading.Thread(target=capture_and_save,
                                      name="CamThread" + str(cam + 1),
                                      args=(cam_cap, cam, dir_to_write,
                                            log_writer,
                                            rgb_stream[cam], depth_stream[cam], io,))
        cam_threads.append(one_thread)
        one_thread.daemon = True
        one_thread.start()

        try:
            while True:
                pass
                # cv2.waitKey(1)
        except KeyboardInterrupt:
            # Stop everything
            for each_thread in cam_threads:
                each_thread.do_run = False
                each_thread.join(1)
            cam_cap.stop_rgb(rgb_stream)
            cam_cap.stop_depth(depth_stream)

            # Stop and quit
            openni2.unload()
            cv2.destroyAllWindows()

if __name__ == '__main__':
    main()      

So, my issue is that if I remove the cv2.imshow() lines from the code, everything runs as expected and I get both camera outputs saved to file. However, with the cv2.imshow() lines, there are only "blank" windows being created and the threads seem to be "stuck", with no output at all.
I have tried several suggestions, including moving the namedWindow creation to the main thread as well as into the capture_and_save thread. I have also tried moving around the waitKey() because it was said that OpenCV only allows waitKey() in the main thread. There was no difference, however.

like image 875
Kanishka Ganguly Avatar asked Oct 17 '22 10:10

Kanishka Ganguly


1 Answers

I solved the issue by using mutables, passing a dictionary cam_disp = {} to the thread and reading the value in the main thread. cv2.imshow() works best when kept in the main thread, so this worked perfectly. I am not sure if this is the "right" way to do this, so all suggestions are welcome.

like image 178
Kanishka Ganguly Avatar answered Oct 22 '22 10:10

Kanishka Ganguly