Dense optical flow with masking

Question

I need to compute dense optical flow of vehicles between two consecutive frames recorded from an overhead camera. I am using an object detector for vehicle detection and Farneback algorithm for optical flow estimation in Python. Presently, the optical flow is computed for the entire image. However, I need to create a mask around the vehicles detected and compute dense optical flow only around the selected region. This will help to speed up optical flow estimation significantly. I can use any other form of dense optical flow estimation too, but not sparse optical flow (e.g., Lucas-Kanade method). One option can be to extract vehicle regions and input each vehicle region individually for optical flow estimation, but I am looking for a better solution.

flow = cv2.calcOpticalFlowFarneback(prvs,next, None, 0.5, 3, 15, 3, 5, 1.2, 0)

The expected optical flow result will be zero for non-vehicle masked region without any optical flow computation.

Answer 1

Unfortunately, acceleration the motion estimation with as you plan by masking or only providing the bounding boxes of your detector will be no good idea. If you use a dense optical flow approach you need to provide the whole image and compute the flow field for the whole image. If you overlay the image with by removing you will introduce artifical edges which will confuse the method. On the otherhand, input each vehicle region individually you will lose the ability of the dense flow method to compute larger(fast) motions. Methods such as farneback uses image pyramids to deal with large motion, i.e. they are based on downscaling the input image. If the regions are to small than this limits the ability to compute large motions. A possibility to reduce the runtime would be to apply a Sparse method such as (Lucas Kanade or RLOF) and apply the SparseToDense interpolation (e.g. EdgeAwareInterpolation in OpenCV) to your vehicle region and get the dense motion field for that area.

Answer 2

This can be a healthy approach for speed up, for some use-cases of optical flow.

To be sure we are on the same page, the optical flow will not directly be used as the main source of information for tracking the car. That is handled by another module... probably some object detector based on deep learning. This object detector is likely intense computation-wise (unless you got some special hardware for it, like a dedicated GPU or VPU).

The dense flow can be a great compliment, to get low-latency information about what is happening to the car and around it, and maybe be used as part of bootstrapping the later detector (providing a prior to the detector in the successive frames, as in a Kalman filter tracking sort of setting).

With this assumption, you can easily optimize your optical flow calculations as you mention. I would suggest you skip the notion of a dense mask here, just use the bounding box from your detector. This will be your region of interest for the flow calculation. If you are in frame t, where your detection is at, and want to use optical flow from 2 frames, t and t-1, then use the same bounding box for both frames. This will give you 2 ROIs, and those are fed to your dense optical flow module.

Each optical flow algorithm has support regions for their low-level operations, and they dont all handle image boundaries well. Therefore, make sure you supply ample extra space to your bounding boxes (as far as they can fit in the original video frames).

For clarity, if you have bounding box for car detection as bbox(x,y,w,h), you want a good margin m, so that the used bounding box for dense flow is bbox(x-m/2,y-m/2,w+m,h+m). The margin is something you want to set depending on your optical flow method (and its parameters).

edit: code for you here, you go. Note, I havent tested this at all:

// This is c++, since we are talking optimizations
// I am uncertain of they python wrappers, but below I am fairly certain
// will not make any deep copies outside of the flow module.
// Assume you have INew and IOld, the two frames of the video:
// assume your bounding box from your detector is a cv::Rect b
float m = 4.0; // 4 pixels as an example margin...
Rect boxFlow(b.x - m/2, b.y - m/2, b.width + m, b.height + m);
Mat roiNew(INew (boxFlow));
Mat roiOld(IOld (boxFlow));
Mat uflow;
calcOpticalFlowFarneback(roiOld, roiNew, uflow, 0.5, 3, 15, 3, 5, 1.2, 0);

However, depending on your setup, this may not be a significant speed up in the end. For above code, I dont know how memory is handled inside the flow module, and in your python version, I am even less certain how it will be handled.

Consider that in a streamlined dense optical flow approach (where you use same size data througout), you have the same size of your memory buffers in your pipeline. A naive implementation of the approach you mention would need to allocate memory dynamically, as the size and number of your bounding boxes will vary over time.