After having calculate SIFT or ORB on a frame, how to real time track the object in a video?

Question

Bascially I want to detect an object and than track it in a video (frame-by-frame).

I can detect it on the first frame with for example ORB or SIFT. But for the next frames (or say next XX frames) I would like to avoid to calulcate again all the keypoints (ORB or SIFT) to detect it again.

Considering I want to track it in a video real time, what could I do ?

Answer 1

A common option is using a patchtracker. That means that you just search for keypoints in an area of, for example, 8 pixels around the previous frame keypoint. You can perform cv::matchTemplate() of an area surrounding the keypoint, instead of using SIFT.

Performing a pyramidal search helps to improve frame-rate. You first search at a lower scale, if you cannot find the keypoint you double the scale.

If patchtracker fails, because the image moves too fast, you just have to reinitialize the system by applying SIFT again. I would use FAST instead of SIFT. You can use SIFT for the marker, and then FAST for detecting keypoints real-time, generating SIFT descriptors.

Answer 2

Detecting and tracking object in a video is a very large topic and the way to go highly depends on your application. There is no magic bullet! If you achieve the detection part, you can try tracking by meanshift on color (maybe HSV color space) likelihood if the object you need to track is colored .. , or try template matching, or .. You need to be more specific on your needs.