opencv background substraction

Question

I have an image of the background scene and an image of the same scene with objects in front. Now I want to create a mask of the object in the foreground with background substraction. Both images are RGB.

I have already created the following code:

cv::Mat diff;
diff.create(orgImage.dims, orgImage.size, CV_8UC3);
diff = abs(orgImage-refImage);

cv::Mat mask(diff.rows, diff.cols, CV_8U, cv::Scalar(0,0,0));
//mask = (diff > 10);

for (int j=0; j<diff.rows; j++) {
    // get the address of row j
    //uchar* dataIn= diff.ptr<uchar>(j);
    //uchar* dataOut= mask.ptr<uchar>(j);
    for (int i=0; i<diff.cols; i++) {
        if(diff.at<cv::Vec3b>(j,i)[0] > 30 || diff.at<cv::Vec3b>(j,i)[1] > 30 || diff.at<cv::Vec3b>(j,i)[2] > 30)
            mask.at<uchar>(j,i) = 255;
    }
}

I dont know if I am doing this right?

Answer 1

Have a look at the inRange function from OpenCV. This will allow you to set multiple thresholds at the same time for a 3 channel image.

So, to create the mask you were looking for, do the following:

inRange(diff, Scalar(30, 30, 30), Scalar(255, 255, 255), mask);

This should also be faster than trying to access each pixel yourself.

EDIT : If skin detection is what you are trying to do, I would first do skin detection, and then afterwards do background subtraction to remove the background. Otherwise, your skin detector will have to take into account the intensity shift caused by the subtraction.

Check out my other answer, about good techniques for skin detection.

EDIT :

Is this any faster?

int main(int argc, char* argv[])
{
    Mat fg = imread("fg.jpg");
    Mat bg = imread("bg.jpg");

    cvtColor(fg, fg, CV_RGB2YCrCb);
    cvtColor(bg, bg, CV_RGB2YCrCb);

    Mat distance = Mat::zeros(fg.size(), CV_32F);

    vector<Mat> fgChannels;
    split(fg, fgChannels);

    vector<Mat> bgChannels;
    split(bg, bgChannels);

    for(size_t i = 0; i < fgChannels.size(); i++)
    {
        Mat temp = abs(fgChannels[i] - bgChannels[i]);
        temp.convertTo(temp, CV_32F);

        distance = distance + temp;
    }


    Mat mask;
    threshold(distance, mask, 35, 255, THRESH_BINARY);

    Mat kernel5x5 = getStructuringElement(MORPH_RECT, Size(5, 5));
    morphologyEx(mask, mask, MORPH_OPEN, kernel5x5);

    imshow("fg", fg);
    imshow("bg", bg);
    imshow("mask", mask);

    waitKey();

    return 0;
}

This code produces this mask based on your input imagery:

Finally, here is what I get using my simple thresholding method:

    Mat diff = fgYcc - bgYcc;
    vector<Mat> diffChannels;
    split(diff, diffChannels);

    // only operating on luminance for background subtraction...
    threshold(diffChannels[0], bgfgMask, 1, 255.0, THRESH_BINARY_INV);

    Mat kernel5x5 = getStructuringElement(MORPH_RECT, Size(5, 5));
    morphologyEx(bgfgMask, bgfgMask, MORPH_OPEN, kernel5x5);

This produce the following mask:

Answer 2

I think when I'm doing it like this I get the right results: (in the YCrCb colorspace) but accessing each px is slow so I need to find another algorithm

    cv::Mat mask(image.rows, image.cols, CV_8U, cv::Scalar(0,0,0));

    cv::Mat_<cv::Vec3b>::const_iterator itImage= image.begin<cv::Vec3b>();
    cv::Mat_<cv::Vec3b>::const_iterator itend= image.end<cv::Vec3b>();
    cv::Mat_<cv::Vec3b>::iterator itRef= refRoi.begin<cv::Vec3b>();
    cv::Mat_<uchar>::iterator itMask= mask.begin<uchar>();

    for ( ; itImage!= itend; ++itImage, ++itRef, ++itMask) {
        int distance = abs((*itImage)[0]-(*itRef)[0])+
                        abs((*itImage)[1]-(*itRef)[1])+
                        abs((*itImage)[2]-(*itRef)[2]);

        if(distance < 30)
            *itMask = 0;
        else
            *itMask = 255;
    }