Android Asynctask to process live video frames

Question

I am using OpenCV to attempt to do some live video processing. Since the processing is fairly heavy, it delays the output frames significantly, making the live stream look choppy.

I'd like to offload some of the processing into an AsyncTask. I've tried it and it actually makes the video much smoother. However, it ends up starting a large amount of Tasks at once, and then they will slowly start returning with some results.

Is there any way to slow this down, and wait for a result, either by using Synchronize statements, or some other method?

On each camera frame, I start one of these tasks. DoImgProcessing does the long processing and returns a string result.

private class LongOperation extends AsyncTask<Mat, Void, String> {

    @Override
    protected String doInBackground(Mat... params) {
        Mat inputFrame = params[0];
        cropToCenter(inputFrame);
        return doImgProcessing(inputFrame);
    }      

    @Override
    protected void onPostExecute(String result) {
        Log.d(TAG, "on post execute: "+result);

    }

    @Override
    protected void onPreExecute() {
        Log.d(TAG, "on pre execute");
    }
}

public Mat onCameraFrame(Mat inputFrame) {
    inputFrame.copyTo(mRgba);//this will be used for the live stream
    LongOperation op = new LongOperation();
    op.execute(inputFrame);
    return mRgba;
}

Answer 1

I would do something like that :

// Example value for a timeout.
private static final long TIMEOUT = 1000L;

private BlockingQueue<Mat> frames = new LinkedBlockingQueue<Mat>();

Thread worker = new Thread() {
    @Override
    public void run() {
        while (running) {
            Mat inputFrame = frames.poll(TIMEOUT, TimeUnit.MILLISECONDS);
            if (inputFrame == null) {
                // timeout. Also, with a try {} catch block poll can be interrupted via Thread.interrupt() so not to wait for the timeout.
                continue;
            }
            cropToCenter(inputFrame);
            String result = doImgProcessing(inputFrame);
        }
    }
};
worker.start();

public Mat onCameraFrame(Mat inputFrame) {
    inputFrame.copyTo(mRgba);//this will be used for the live stream
    frames.put(inputFrame);
    return mRgba;
}

The onCameraFrame puts the frame on the Queue, the worker Thread polls from the Queue.

This decorelate the reception and the treatment of the frame. You can monitor the growth of the Queue using frames.size().

This is a typical producer-consumer example.

Answer 2

If you're doing this on each frame, it sounds like you need a thread instead. An AsyncTask is for when you want to do a one-off activity on another thread. Here you want to do it repeatedly. Just create a thread, and when it finishes a frame have it post a message to a handler to run the post step on the UI thread. It can wait on a semaphore at the top of its loop for the next frame to be ready.