Known object detection performance improvement with OpenCV

Question

I am working on a project where I have a to detect a known picture in a scene in "real time" in a mobile context (that means I'm capturing frames using a smartphone camera and resizing the frame to be 150x225). The picture itself can be rather complex. Right now, I'm processing each frame in 1.2s in average (using OpenCV). I'm looking for ways to improve this processing time and global accuracy. My current implementation work as follow :

1. Capture the frame
2. Convert it to grayscale
3. Detect the keypoint and extract the descriptors using ORB
4. Match the descriptor (2NN) (object -> scene) and filter them with ratio test
5. Match the descriptor (2NN) (scene -> object) and filter them with ratio test
6. Non-symmetrical matching removal with 4. and 5.
7. Compute the matching confidence (% of matched keypoints against total keypoints)

My approach might not be the right one but the results are OK even though there's a lot of room for improvement. I already noticed that SURF extraction is too slow and I couldn't manage to use homography (it might be related to ORB). All suggestions are welcome!

Answer 1

Performance is always an issue on mobiles :)

There are a few things you can do. OpenCV: C++ and C performance comparison explains generic methods on processing time improvements.

And some specifics for your project:

• If you capture color images and the convert them to grayscale, that is a biig waste of resources. The native camera format is YUV. It gets converted to RGB, which is costly, then to gray, which again is costly. All this while the first channel in YUV (Y) is the grayscale... So, capture YUV, and extract the first channel by copying the first part of the image data (YUV on Android is planar, that means that the first w*h pixels belong to the Y channel)
• ORB was created to be fast. And it is. But just a few weeks ago FREAK was added to OpenCV. That is a new descriptor, whose authors claim is both more accurate and faster than ORB/SIFT/SURF/etc. Give it a try.YOu can find it in opencv >= 2.4.2 (This is the current now)

EDIT

Brad Larsen question is illuminating - if the matcher stays 900ms to process, then that's a problem! Check this post by Andrey Kamaev How Does OpenCV ORB Feature Detector Work? where he explains the possible combinations between descriptors and matchers. Try the FLANN-based uchar matcher.

And also, I suppose you get an awful lot of detections - hundreds or thousands - if it takes that much to match them. Try to limit the detections, or select only the first n best values.

Answer 2

You should try FAST to detect the object in the scene, is faster than SURF and you can find articles that use a pyramidal version of FAST. To improve performance on mobiles you can optimize loops, use fixed-poit arithmetics, etc. Good luck.