Remove background and get deer as a fore ground?

Question

I want to remove background and get deer as a foreground image.

This is my source image captured by trail camera:

This is what I want to get. This output image can be a binary image or RGB.

I worked on it and try many methods to get solution but every time it failed at specific point. So please first understand what is my exact problem.

1. Image are captured by a trail camera and camera is motion detector. when deer come in front of camera it capture image.
2. Scene mode change with respect to weather changing or day and night etc. So I can't use frame difference or some thing like this.
3. Segmentation may be not work correctly because Foreground (deer) and Background have same color in many cases.

If anyone still have any ambiguity in my question then please first ask me to clear and then answer, it will be appreciated. Thanks in advance.

Answer 1

Here's what I would do:

As was commented to your question, you can detect the dear and then perform grabcut to segment it from the picture.

To detect the dear, I would couple a classifier with a sliding window approach. That would mean that you'll have a classifier that given a patch (can be a large patch) in the image, output's a score of how much that patch is similar to a dear. The sliding window approach means that you loop on the window size and then loop on the window location. For each position of the window in the image, you should apply the classifier on that window and get a score of how much that window "looks like" a dear. Once you've done that, threshold all the scores to get the "best windows", i.e. the windows that are most similar to a dear. The rational behind this is that if we a dear is present at some location in the image, the classifier will output a high score at all windows that are close/overlap with the actual dear location. We would like to merge all that locations to a single location. That can be done by applying the functions groupRectangles from OpenCV:

http://docs.opencv.org/modules/objdetect/doc/cascade_classification.html#grouprectangles

Take a look at some face detection example from OpenCV, it basically does the same (sliding window + classifier) where the classifier is a Haar cascade.

Now, I didn't mention what that "dear classifier" can be. You can use HOG+SVM (which are both included in OpenCV) or use a much powerful approach of running a deep convulutional neural network (deep CNN). Luckily, you don't need to train a deep CNN. You can use the following packages with their "off the shelf" ImageNet networks (which are very powerful and might even be able to identify a dear without further training):

Decaf- which can be used only for research purposes: https://github.com/UCB-ICSI-Vision-Group/decaf-release/

Or Caffe - which is BSD licensed:

http://caffe.berkeleyvision.org/

There are other packages of which you can read about here: http://deeplearning.net/software_links/

The most common ones are Theano, Cuda ConvNet's and OverFeat (but that's really opinion based, you should chose the best package from the list that I linked to).

The "off the shelf" ImageNet network were trained on roughly 10M images from 1000 categories. If those categories contain "dear", that you can just use them as is. If not, you can use them to extract features (as a 4096 dimensional vector in the case of Decaf) and train a classifier on positive and negative images to build a "dear classifier".

Now, once you detected the dear, meaning you have a bounding box around it, you can apply grabcut:

http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_grabcut/py_grabcut.html

You'll need an initial scribble on the dear to perform grabcu. You can just take a horizontal line in the middle of the bounding box and hope that it will be on the dear's torso. More elaborate approaches would be to find the symmetry axis of the dear and use that as a scribble, but you would have to google, research an implement some method to extract symmetry axis from the image.

That's about it. Not straightforward, but so is the problem.

Please let me know if you have any questions.