OpenCV image processing in Android service

Question

My android app does image processing with opencv for android in an activity with a JavaCameraView. That works fine. Now I want to do the same image processing in background without any preview for the user. I started with an android service.

With this code I can load OpenCV succesful in a service:

import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;

import android.app.Service;
import android.content.Intent;
import android.os.IBinder;
import android.util.Log;

public class CameraService extends Service {

private static final String TAG = "CameraService";

BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(CameraService.this) {

@Override
public void onManagerConnected(int status) {
        switch (status) {
        case LoaderCallbackInterface.SUCCESS: {
            Log.i("", "OpenCV loaded successfully");
        }
            break;
        default: {
            super.onManagerConnected(status);
        }
            break;
        }
    }
};

@Override
public int onStartCommand(Intent intent, int flags, int startId) {
    if(OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_4,
                getApplicationContext(), mLoaderCallback)) {
        Log.i(TAG, "Loaded OpenCV");
    }
    else
        Log.i(TAG, "Couldn't load OpenCV");
    return super.onStartCommand(intent, flags, startId);
}

@Override
public IBinder onBind(Intent intent) {
    return null;
}

}

But I don't know how to grab frames like in my former activity onCameraFrame()? There I have implentent the CvCameraViewListener2 but not in my service, because it requires a CameraBridgeViewBase which I don't want to show anymore. How can I do image processing in a background like this?

UPDATE -> 2

I added a runnable to grab frames like you told me. Loading OpenCV and connecting to camera works fine now. But before grabbing any frames, he skips frames and aborts, because the application is doing too much work on main thread.

Thats my whole camera service right now:

public final class MyService extends Service {

private static final String TAG = MyService.class.getSimpleName();
private boolean mStopThread;
private Thread mThread;
private VideoCapture mCamera;
private int mCameraIndex = -1;

BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
        case LoaderCallbackInterface.SUCCESS: {
            Log.i("", "OpenCV loaded successfully");

            try {
                if (!connectCamera(640, 480))
                    Log.e(TAG, "Could not connect camera");
                else
                    Log.d(TAG, "Camera successfully connected");
            } catch (Exception e) {
                Log.e(TAG, "MyServer.connectCamera throws an exception: " + e.getMessage());
            }

        }
            break;
        default: {
            super.onManagerConnected(status);
        }
            break;
        }
    }
};

@Override
public int onStartCommand(Intent intent, int flags, int startId) {
    if(OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_4, getApplicationContext(), mLoaderCallback))
        Log.i(TAG, "Loaded OpenCV");
    else
        Log.i(TAG, "Couldn't load OpenCV");
    return super.onStartCommand(intent, flags, startId);
}

public void onDestroy() {
    this.disconnectCamera();
    Log.d(TAG, "onDestroy");
    super.onDestroy();
}

private boolean connectCamera(int width, int height) {
    /* First step - initialize camera connection */
    if (!initializeCamera(width, height)) {
        Log.d(TAG, "initializeCamera failed");
        return false;
    } else {
        Log.d(TAG, "initializeCamera successfully");
    /* start update thread */
    mThread = new Thread(new CameraWorker());
    mThread.start();

    return true;
    }
}

private boolean initializeCamera(int width, int height) {
    synchronized (this) {
        if (mCameraIndex == -1)
            mCamera = new VideoCapture(Highgui.CV_CAP_ANDROID);
        else
            mCamera = new VideoCapture(Highgui.CV_CAP_ANDROID + mCameraIndex);

        if (mCamera == null)
            return false;

        if (mCamera.isOpened() == false)
            return false;

        /* Select the size that fits surface considering maximum size allowed */
        Size frameSize = new Size(width, height);
        mCamera.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, frameSize.width);
        mCamera.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, frameSize.height);
    }

    return true;
}

private void releaseCamera() {
    synchronized (this) {
        if (mCamera != null) {
            mCamera.release();
        }
    }
}

private void disconnectCamera() {
    // 1. Stop thread which updating the frames
    // 2. Stop camera and release it
    try {
        mStopThread = true;
        mThread.join();
    } catch (InterruptedException e) {
        e.printStackTrace();
    } finally {
        mThread =  null;
        mStopThread = false;
    }
    releaseCamera();
}

private class CameraWorker implements Runnable {
    public void run() {
        do {
            if (!mCamera.grab()) {
                Log.e(TAG, "Camera frame grab failed");
                break;
            }
            Log.e(TAG, "Camera frame grabbed");
            // img processing
        } while (!mStopThread);
    }
}

@Override
public IBinder onBind(Intent intent) {
    return null; // Not used
}
}

My log:

11-29 12:28:24.370: D/OpenCVManager/Helper(5257): Init finished with status 0
11-29 12:28:24.370: D/OpenCVManager/Helper(5257): Unbind from service
11-29 12:28:24.380: D/OpenCVManager/Helper(5257): Calling using callback
11-29 12:28:24.380: I/(5257): OpenCV loaded successfully
11-29 12:28:24.380: D/OpenCV::camera(5257): CvCapture_Android::CvCapture_Android(0)
11-29 12:28:24.440: D/OpenCV_NativeCamera(5257): Connecting to CameraService v 3D
11-29 12:28:24.670: D/OpenCV_NativeCamera(5257): Instantiated new CameraHandler (0x75e4f29d, 0x71e178b8)
11-29 12:28:24.750: D/OpenCV_NativeCamera(5257): Starting preview
11-29 12:28:25.421: E/OpenCV_NativeCamera(5257): CameraHandler::doCall(void*, size_t): cameraCallback returns false (camera connection will be closed)
11-29 12:28:25.421: E/BufferQueue(5257): [unnamed-5257-0] dequeueBuffer: min undequeued buffer count (2) exceeded (dequeued=11 undequeudCount=0)
11-29 12:28:25.431: E/BufferQueue(5257): [unnamed-5257-0] dequeueBuffer: min undequeued buffer count (2) exceeded (dequeued=10 undequeudCount=1)
11-29 12:28:25.451: D/OpenCV_NativeCamera(5257): Preview started successfully
11-29 12:28:25.451: D/OpenCV_NativeCamera(5257): CameraHandler::setProperty(0, 640.000000)
11-29 12:28:25.451: D/OpenCV_NativeCamera(5257): CameraHandler::setProperty(1, 480.000000)
11-29 12:28:25.451: D/MyService(5257): initializeCamera successfully
11-29 12:28:25.451: D/MyService(5257): Camera successfully connected
11-29 12:28:25.451: I/Choreographer(5257): Skipped 86 frames!  The application may be doing too much work on its main thread.
11-29 12:28:25.471: A/libc(5257): @@@ ABORTING: LIBC: HEAP MEMORY CORRUPTION IN tmalloc_small
11-29 12:28:25.471: A/libc(5257): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 5257 ()

Whats wrong and what can I do now?

Answer 1

You can use native variant(VideoCapture). CameraBridgeViewBase - extends from Android.Camera and too dont't work in background. If you don't find example, in OpenCv-2.4.2 Android library see example FaceDetection

UPD:

You can get frames from camera, using Intefrace Runnable:

private VideoCapture        mCamera;

public void run() {
        Log.i(TAG, "Starting processing thread");

    while (true) {
        Bitmap bmp = null;

        synchronized (this) {
            if (mCamera == null)
                break;

            if (!mCamera.grab()) {
                Log.e(TAG, "mCamera.grab() failed");
                break;
            }

            bmp = processFrame(mCamera);

        }

}