Feature tracking using optical flow

Question

I found a similar question in the forum. But the answer in there does not answer my question.

• If I do feature detection (goodFeaturesToTrack) only once on the first image, and then use optical flow (calcOpticalFlowPyrLK) to track these features, the problem is: only the features detected on the first image can be tracked. When these features go beyond of the image, there would be no features to track.

• If I do feature detection for every new image, the feature tracking is not stable, because the feature detected last time may not be detected this time.

I am using optical flow for 3D reconstruction. So I'm not interested in tracking what features, instead, I only care whether features in the field of view can be tracked stably. To summarize, my question is: how can I use optical flow to track old features, and in the meantime add new image features that come into the field of view and remove old features that go beyond the field of view?

Answer 1

Several approaches are possible. A good method goes like this:

1. in Frame 1 detect N features, this is the Keyframe m=1
2. in Frame k track the features by optical flow
3. in Frame k if the number of successfully tracked features drops under N/2:
   • this frame is the keyframe m+1
   • compute the homography or the fundamental matrix describing the motion between the keyframes m and m+1
   • detect N features and discard the old ones
   • k := k+1 go to 2

In this method basically you estimate the camera motion between the last two keyframes.

Since you didn't mention what approach is used for 3D reconstruction I assumed either H or F are computed to estimated motion first. To estimate them accurately the baseline between the keyframes should be as wide as possible. In general, the best strategy is take into account the rough motion model of the camera. If the camera is held by hand a different strategy should be used compared to when the camera is fixed on the top of a car or a robot. I can provide a minimal working example in Python if that helps, let me know.

Answer 2

Just for documentation purposes, there are several good GPU / C++ implementations of optical flow tracking. Your code may be better for your purposes, but if all you need is the output data of the tracks, consider checking any of the following sources: here, here, or here.