How can I compute optical flow from a depth image stream from a depth camera?

Question

I have a depth camera feed already set up and in order to make it more interesting I want to compute some data out of it like normals, motion/optical flow and other data sets to use them for visual effects. I am particularly interested in optical flow and whether it can be computed from a depth only stream.

Has this been implemented? If so I'd like to know what are the methods and understand which one would be the easiest to use.

Answer 1

I worked on Kinect depth camera and implemented a patient tracking algorithm. The algorithm itself is commercial and I cannot disclose the details. But I can give my two cents here.

1. The depth feed from Kinect should not directly used for optical flow (motion tracking), due to no depth pixels. You can use inpainting to fill in gaps in the depth image. If you are using OpenCV, you can refer to the implementation here.

http://www.morethantechnical.com/2011/03/05/neat-opencv-smoothing-trick-when-kineacking-kinect-hacking-w-code/

2. I suggest using a smoothing filter after inpainting to have a smooth depth data near object edges. You can use simple filters present in OpenCV with depth stream. It would be nice to downsample 16 bit depth to 8 bit RGB image to help visualize disparity image.

3. I believe you can then use the resulting stream with optical flow algorithm from OpenCV. Here is an example.

http://docs.opencv.org/3.1.0/d7/d8b/tutorial_py_lucas_kanade.html#gsc.tab=0

You can also use Dense trajectory implementation, but I believe it is processor intensive and the final frame rate might be really slow.

https://lear.inrialpes.fr/people/wang/dense_trajectories

Hope this helps.