Flask freezes with OpenCV

Question

In my current flask project, I am trying to stream two live videos, which sound’s easy. The problem is that I have only one video source (camera). The purpose is to have two video streams: one without any modifications and one with face detection applied. If user want’s to have face detection, then by clicking the button camera view for him will be changed to the stream, which has face detection applied. If user don’t want to have it, then he will see stream without it. What’s very important - multiple users can view streams at one time. Whole program works on RPi 4B 4gb.

I have a CamerasPool class:

from .CameraHandler import CameraHandler

import cv2

class CamerasPool:
    def __init__(self):
        self.__cameras = []

    def registerCamera(self, id, detection):
        self.__cameras.append(CameraHandler(id, cv2.VideoCapture(0), detection))
        print('Camera registered')

    def getCamWithParameters(self, detection):
        camera = None

        for cam in self.__cameras:
            if cam.getDetectionState() == detection:
                camera = cam
                break

        return camera

And CamerasHandler class:

import cv2
from time import sleep

class CameraHandler():
    def __init__(self, id, cam, detectionState):
        self.__id = id
        self.__cam = cam
        self.__current_frame = None
        self.__shouldDetect = detectionState
        print(f'Camera created with id {id} created')

    def __del__(self):
        self.__cam.release()

    def getDetectionState(self):
        return self.__shouldDetect

    def __detectFace(self, img):
        faces, confidences = cv.detect_face(img)

        for face in faces:
            (startX, startY) = face[0], face[1]
            (endX, endY) = face[2], face[3]

            cv2.rectangle(img, (startX, startY), (endX, endY), (0, 255, 0), 2)

        return img

    def __getFrame(self):
        rval, frame = self.__cam.read()

        if rval:
            frame = cv2.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)

            try:
                if self.__shouldDetect:
                    frame = self.__detectFace(frame)
            except:
                print('Face detection exception')

            (flag, encodedImage) = cv2.imencode(".jpg", frame)

            self.__current_frame = bytearray(encodedImage)

    def gen(self):
        while True:
            self.__getFrame()

            yield (b'--frame\r\n'
                b'Content-Type: image/jpeg\r\n\r\n' + self.__current_frame + b'\r\n')

I am trying to create cameras as follows:

# Create two cameras
print('Before cameras creation')
camerasPool = CamerasPool()
print('After cameras pool creation')
camerasPool.registerCamera(0, False)
camerasPool.registerCamera(1, True)
print('Created both cameras')

As you can see in CamerasPool class I am creating cameras object like this self.__cameras.append(CameraHandler(id, cv2.VideoCapture(0), detection)), which creates two objects which want’s to access the same resource - camera.

When I am launching entire program I can see the following output:

 * Serving Flask app "server/" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://192.168.0.151:7070/ (Press CTRL+C to quit)
 * Restarting with stat
Before cameras creation
After cameras pool creation
 * Debugger is active!
 * Debugger PIN: 196-285-435
Before cameras creation
After cameras pool creation

Program freezes and that’s it. In the output I should see also Camera created with id 0 created and Camera created with id 1 created, but as far as I understand they haven’t been created - program freezes at this steep. I guess that the problem is because of two VideoCapture objects. Can someone help me how to solve this problem? Maybe some other solution how to have two streams from one camera?

Best regards, Piotr

Answer 1

You can't instance two CamerasPool objects since you only have one camera. But what you can do is:

• Change the implementation of CamerasPool so that it becomes its own thread and doesn't block the execution of the python application. The purpose of this class is to simply read frames from the camera, make a copy of each frame and put() them in two separate queue objects. The queues should be global in the program so that they are accessible to the other threads that will need to run concurrently to process the data and stream them.
• Create a new class VideoStream to be responsible to get() frames from a particular queue, process it and stream it. Processing means anything you'd like to do with the frame before it gets streamed to the network: convert to grayscale, draw rectangles, detect faces, etc. This class will also need to run in a separate thread and the constructor will need to receive two parameters: the first to indicate which of the two global queues it should use; the second parameter to indicate whether the frames should be processed before they are streamed;

If you are looking for code examples on how to use multithreading to retrieve frames from a camera and store them in a queue, check this answer, specially the section that says:

How to capture at the closest maximum fps supported by the camera? A threading and queue example.

Keep in mind that your application will have 4 threads:

• the main thread;
• the one that belongs to CamerasPool;
• and the other 2 threads are from each of the VideoStream objects that will be instanced;