Live stream video processing in android using opencv

Question

How do we perform certain basic image processing techniques on a live stream of video frames from a camera, in android using opencv (Opencv4Android) ?

Answer 1

With that little information there is not much we can do to help. However, I will post some code for performing a simple image processing technique to a camera stream: converting the frames to grayscale.

You need to declare the following:

private CameraBridgeViewBase mOpenCvCameraView;
public CvCameraViewListener2 camListener;

Then, you need to initialize OpenCV. You need to add the following block of code before you call your onCreate() method:

private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
        case LoaderCallbackInterface.SUCCESS:
        {
            Log.i("TAG", "OpenCV loaded successfully");
            mOpenCvCameraView.enableView();
            System.loadLibrary("nameOfYourNativeLibrary"); // if you are working with JNI
            run();
        } break;
        default:
        {
            super.onManagerConnected(status);
        } break;
        }
    }
};

You can call

if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback)){
    Log.e("ERR", "Cannot connect to OpenCV Manager");
}else Log.i("SUCCESS", "opencv successfull");

inside your onCreate() method, but what you really need is to override the onResume() method:

@Override
public void onResume()
{
    super.onResume();
    OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback);
    mOpenCvCameraView.enableView();
}

mOpenCvCameraView is a JavaCameraView. Just in case you are not familiar with any of this, this is the xml layout of your MainActivity:

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    xmlns:opencv="http://schemas.android.com/apk/res-auto" 
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:paddingBottom="@dimen/activity_vertical_margin"
    android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    tools:context="com.example.yourapp.MainActivity$PlaceholderFragment" >

<org.opencv.android.JavaCameraView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:visibility="visible"
    android:id="@+id/HelloOpenCvView"
    opencv:show_fps="false"
    opencv:camera_id="any" />

</RelativeLayout>

So your onCreate() method should look like this:

@Override
public void onCreate(Bundle savedInstanceState) {
    Log.i("APP", "called onCreate");
    super.onCreate(savedInstanceState);
    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
    setContentView(R.layout.fragment_main);
    mOpenCvCameraView = (CameraBridgeViewBase)findViewById(R.id.HelloOpenCvView);
    if (!OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback)){
        Log.e("OPENCV", "Cannot connect to OpenCV Manager");
    }else Log.i("OPENCV", "opencv successfull");
}

It is in the missing run() method where you should add the image processing you want to perform. In this case we will convert the frames to grayscale and display them:

private void run(){
    camListener = new CvCameraViewListener2() {

        @Override
        public void onCameraViewStopped() {
            // TODO Auto-generated method stub
        }

        @Override
        public void onCameraViewStarted(int width, int height) {
            // TODO Auto-generated method stub          
        }

        @Override
        public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
            rgb = inputFrame.rgba();
            // here you could just return inputframe.gray(), but for illustration we do the following
            Mat gray = new Mat();
            Imgproc.cvtColor(rgb, gray, Imgproc.COLOR_RGB2GRAY);
            return gray;
        }
    };

    mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);
    mOpenCvCameraView.setCvCameraViewListener(camListener);

}

I hope this helps. If you need anything else let me know.

UPDATE If you want to access the pixel values of a Mat object you need something like this:

double mean = 0;
double size = mat.rows()*mat.cols();
for(int k=0; k<mat.rows(); k++){
    for(int j=0; j<mat.cols(); j++){
        mean += mat.get(k, j)[0]/size;
    }
}

Note that in this case my Mat object has a single channel (that means if you try to access mat.get(k, j)[1] you would obtain an Array out of bounds exception. For RGB Mat objects you should be able to access all mat.get(k, j)[0], mat.get(k, j)[1] and mat.get(k, j)[2]