How can I remove noise from this video sequence?

Question

Hello I am trying to do some image processing. I use Microsoft Kinect to detect humans on a room. I get depth data, do some background subtraction work and end up with a video sequence like this when a person enters the scene and walks around:

http://www.screenr.com/h7f8

I put a video so that you can see the behaviour of the noise in the video. Different colors represent different levels of depth. White represents empty. As you can see it is pretty noisy, especially the red noises.

I need to get rid of everything except the human as much as possible. When I do erosion/dilation (using a very big window size) I can get rid of a lot of the noise but I wondered if there are other methods I can use. Especially the red noise in the video is hard to remove using erosion/dilation.

Some notes:

1) A better background subtraction could be done if we knew when there are no humans in the scene but the background subtraction we do is fully automatic and it works even when there are humans in the scene and even when the camera is moved etc. so this is the best background subtraction we can get right now.

2) The algorithm will work on an embedded system, real time. So the more efficient and easy the algorithm the better. And it doesn't have to be perfect. Though complicated signal processing techniques are also welcome (maybe we might use them on another project who does not need embedded, real time processing).

3) I don't need an actual code. Just ideas.

Answer 1

Just my two cents:

If you don't mind using the SDK for that, then you can very easily keep only the person pixels using the PlayerIndexBitmask as Outlaw Lemur shows.

Now you may not want to be dependable on the drivers for that and want to do it in an image processing level. An approach that we had tried in a project and worked pretty good was contour based. We began by a background subtraction and then we detected the largest contour in the image assuming that this was the person (since usually the noise that remained was very small blobs) and we filled that contour and kept that. You could also use some kind of median filtering as a first pass.

Of course, this is not perfect nor suitable in every case and probably there are a lot better methods. But I'm just throwing it out there in case it helps you come up with any ideas.

Answer 2

Take a look at the eyesweb.

It is a platform for designing that supports kinect device and you can apply noise filters on the outputs. It is a very usefull and simple tool for multimodal systems designing.