Optimizing algorithm for segmentation through image substraction

Question

for a project in OpenCV I would like to segment moving objects as good as possible with of course minimal noise.

For this I would like to use an image substraction algorithm. I already have a running program but didn't find a way today to get fair enough results.

I already have the following (grayscale) images given:

IplImage* grayScale;
IplImage* lastFrame;
IplImage* secondLastFrame;
IplImage* thirdLastFrame;

So far I have tried to substract current frames image and the last frame with cvSub(); or with cvAbsDiff(); to get the moving parts.

But unfortunately I still get a lot of noise there (i.e. due to slightly moving trees when it's windy) and if the object that moves is quite big and has a homogenic color (let's say a person in a white or black shirt), the substraction only detects the changes in the image on the left and right side of the person, not on the body itself, so one object is sometimes detected as two objects...

cvAbsDiff(this->lastFrame,grayScale,output);
cvThreshold(output,output,10,250, CV_THRESH_BINARY);
cvErode(output,output, NULL, 2);
cvDilate(output,output, NULL, 2);

To get rid of this noise I tried eroding and dilating the images with cvErode() and cvDilate() but this is quite slow and if the moving objects on the screen are small the erosion deletes quite a bit to much of the object so after delating I don't always get a good result or splitted up objects.

After this I do a cvFindContours() to get contours, check on size and if it fits draw a rectangle around the moving objects. But results are poor because often an object is split into several rectangles due to the bad segmentation.

A friend now told me I might try using more than two following frames for the substraction since this might already reduce the noise... but I don't really know what he meant by that and how I should add/substract the frames to get an image that is almost noise free and shows big enough object blobs.

Can anybody help me with that? How can I use more than one frames to get an image that has as minimum noise as possible but with big enough blobs for the moving objects? I would be thankful for any tipps...

ADDITIONS:

I have uploaded a current video right here: http://temp.tinytall.de/ Maybe somebody wants to try it there...

This is a frame from it: The left image shows my results from cvFindContours() and the right one is the segmented image on which I then try to find the contours...

So one large objects it works fine if they are moving fast enough... i.e. the bicycle.. but on walking people it doesn't always get a good result though... Any ideas?

Answer 1

Given three adjacent frames A, B, C you can get two frame differences X and Y. By combining X and Y (through, e.g. thresholding and then logical AND operation) you can reduce the effect of the noise. An unwanted side effect is that the motion-detected area will be slightly smaller than ideal (the AND operation will reduce the area).

Since image sequence motion estimation has been well researched for decades, you may want to read about more sophisticated methods of motion detection, e.g. working with motion vector fields. Google Scholar is your friend in that case.

Answer 2

It seems like you have a fixed background. One possible solution is to let the computer learn the background, eg. by taking an average over time. Then calculate the difference between the average image and the current. Differences are likely to origin from moving objects.

Answer 3

Well, it is a very dicey subject. Motion estimation is quite complex. So try to find good literature and avoid inventing algorithms :)

My suggestions are:

Search for bundling images for motion estimation. Bundling is using many images to reduce noise and error rate.

Eventually, if you want to be robust, look into what is known as a Kalman filter. If you're tracking objects, you don't want them to make "infinite speed jumps" in between your frames (which is usually noise or misses). This is one C++ lib I strongly suggest Kalman filter

Finally, MonoSLAM, I'm pushing a bit :) Andrew Davison: Research