Parallel image detection and camera preview OpenCV Android

Question

I'm using OpenCV to detect an image. Here is my problem: my function detect_image(mRgba) needs some time to perform operations and give some results. While function is computing camera preview is frozen because it only shows image when code reaches return inputFrame.rgba() I would like to know how to make those operation parallel, function will be computing in a background while camera preview is working with normal speed.

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
mRgba = inputFrame.rgba();
detect_image(mRgba);
return inputFrame.rgba();
}

Answer 1

To just get a taste at parallelization, the simple approach would be to just use an AsyncTask to process your images:

AsyncTask reference page

A more friendly introduction can be found here:

http://android-developers.blogspot.co.il/2010/07/multithreading-for-performance.html

while this:

http://developer.att.com/developer/forward.jsp?passedItemId=11900176

is a nice all-around introduction to multi-threading on Android.

If you want to just get started, a simple algorithm should work like this:

• from within your "onCameraFrame" method check if you have an AsyncThread for processing the image which is already running
• if the answer is "yes", just show mRgba in the preview window and return
• if the answer is "no" start a new AsyncThread and let it run "detectImage" on mRgba, making sure that the results are saved in the onPostExecute method.

With this algorithm, if your system can detect 4 images per second while taking a preview at 60fps (for example), you will be able to get a smooth video with a new result about each 20-30 frames on a single processor device, under the realistic assumption that detect_image is CPU intensive while the camera preview/display are I/O intensive.

   Capture:     x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x....
   Processing:  1.......1.......1.......1.....1.......1....
                time ------------------------------------>

Starting with HoneyComb, a more refined approach would be to account for the number of cores in your CPU (multicore phones/tablets are becoming increasingly common) and start N AsyncTask in parallel (one for each core), feeding a different preview image to each one (maybe using a thread pool...).

If you separate each thread by a fixed delay (about the duration of detectImage/N ), you should get a constant stream of results with a frequency that should be a multiple of the single threaded version.

   Capture:     x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x....
   Processing:  1.2.3.4.1.2.3.4.1.2.3.4.1.2.3.4.1.2.3.4....
                time ------------------------------------>

Hope this helps