cv::goodFeaturesToTrack doesn't return any features

Question

I'm trying to use cv::calcOpticalFlowPyrLK but sometimes an internal assertion in that function fails. The assertion is npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0. I'm using OpenCV 2.3.1. The source code for this function is available here.

It's hard to wrap my head around their code, especially because of my limited experience with computer graphics and their lack of comments. Why is this assertion being triggered and what does it say about my problem?

Edit: I call the function as follows:

cv::calcOpticalFlowPyrLK(curBwFrame, prvFrame, features, newFeatures, trackingStatus, errors);

I found out that the features vector, which was obtained by calling cv::goodFeaturesToTrack(curBwFrame, features, 5, 0.2, 0.5, skinMask); with a non-empty mask that appears to be sufficiently big and a valid image, doesn't contain any features. How can this happen?

curBwFrame

skinMask

I'm able to reproduce the problem using the following code snippet:

#include <vector>
#include <cassert>
#include <opencv2\opencv.hpp>
using std::vector;
using namespace cv;

int main() {
    vector<Point2f> features;
    cv::Mat curBwFrame = imread("curBwFrame.png");
    cv::cvtColor(curBwFrame, curBwFrame, CV_RGB2GRAY);
    imwrite("test.png", curBwFrame);

    cv::Mat skinMask = imread("skinMask.png");
    cv::cvtColor(skinMask, skinMask, CV_RGB2GRAY);
    imwrite("test.png", skinMask);

    cv::goodFeaturesToTrack(curBwFrame, features, 5, 0.2, 0.5, skinMask);
    assert(features.size() > 0);

    return 0;
}

Answer 1

The main problem are your parameters.In the OpenCV 2.3.2 documentation (no compatibility change between 2.3.1) this is the description of the method parameters:

void goodFeaturesToTrack(InputArray image, OutputArray corners, int maxCorners, double qualityLevel, double minDistance, InputArray mask=noArray(), int blockSize=3, bool useHarrisDetector=false, double k=0.04 )

Parameters:

• image – Input 8-bit or floating-point 32-bit, single-channel image.
• corners – Output vector of detected corners.
• maxCorners – Maximum number of corners to return. If there are more corners than are found, the strongest of them is returned.
• qualityLevel – Parameter characterizing the minimal accepted quality of image corners. The parameter value is multiplied by the best corner quality measure, which is the minimal eigenvalue (see cornerMinEigenVal() ) or the Harris function response (see cornerHarris() ). The corners with the quality measure less than the product are rejected. For example, if the best corner has the quality measure = 1500, and the qualityLevel=0.01 , then all the corners with the quality measure less than 15 are rejected.
• minDistance – Minimum possible Euclidean distance between the returned corners.
• mask – Optional region of interest. If the image is not empty (it needs to have the type CV_8UC1 and the same size as image ), it specifies the region in which the corners are detected.
• blockSize – Size of an average block for computing a derivative covariation matrix over each pixel neighborhood. See cornerEigenValsAndVecs() .
• useHarrisDetector – Parameter indicating whether to use a Harris detector (see cornerHarris()) or cornerMinEigenVal().
• k – Free parameter of the Harris detector.

I recommend you to play a little with qualityLevel and minDistance to suffice your needs.

Answer 2

Have you tried goodFeaturesToTrack without a mask to see if it detects features inside the masked region? It is possible that, because the image is dark, and the region is a bit textureless, that goodFeaturesToTrack fails to find features there.

You might also try ORB or FAST instead of goodFeaturesToTrack. I have successfully used ORB with calcOpticalFlowPyrLK (but didn't try to use a mask).

Or you could try to brighten up the image or even enhance the contrast. Not really sure if this brings improvements because I think the biggest problem of your scenario is that objects in the scene do not have enough texture or corners, which are the more suitable features for these detectors. I recommend that you try ORB and see if you get more points.