Hand Detection results in jerky cursor

Question

I've written a program that uses the Depth data from a Kinect, and does blob detection to find a user's hand. However, when using the user's hand to control the mouse, it becomes very jerky, probably because people aren't very good at holding body parts completely still.

I've tried averaging the position based on the last ten positioning's, but that just resulted in lag time without actually preventing jerkiness. The best solution so far that I've used is to not move the cursor if the pixel change is less than 10 in both directions (i.e., a 10 pixel change in either direction results in movement). This is okay, but it's still kinda jerky, and results in a clunky interface because you don't have fine precision.

How can I compensate for the lack of steadiness in the human form so that the mouse isn't so jerky?

Answer 1

This will in any case be a tradeoff between lag and stability.

Check your data. You may find that the jerking is because of low resolution in Kinect. If so the jerking distance will be determined at how close you are to the Kinect cameras. When you are too far away the camera resolution is too low and it will keep bouncing between one or two pixels (stereo cams).

You are thinking in the right direction by calculating average and having a threshold for movement. You say you have calculated average for the last 10 positions, which with a resolution of 30 fps causes a 0,33 second delay.

You may want to average out only the 5 last (experiment), and instead of average calculate the mean value.

Just a thought; movement rarely comes alone, so you could set a threshold for when you decrease the number of samples used for averaging/mean.

Answer 2

What is your sample rate? 10 positions is likely to be just a hundredth of a second. You may want to average the last 10th or 3rd of a second.

Answer 3

Did you try to apply a median filter to the depth map before doing your blob detection? I used that in a finger tracking demo and it greatly improved the steadiness.

A bandwidth between 3 and 5 gave me the best results (5 kills a bit the fps but it's really smooth).