Process video frame by frame in OpenCV on Android

Question

My goal is as follows: I have to read in a video that is stored on the sd card, process it frame for frame and then store it in a new file on the SD card again. The problem is that OpenCV4Android does not come with an video encoder/decoder as it does not include ffmpeg. Moreover, using JavaCV for processing the image is no option for me, as the code is already written in native OpenCV and I access them through the JNI. I did a lot of reading here on stackoverflow and the rest of Google. But I did not find the solution.

1. JavaCV allows me to read a video frame by frame and also store it frame by frame. However, I am not able to convert the video to plain OpenCV Mat objects which can be processed by usual OpenCV4Android.

2. I read about JCodec as a library for encoding/decoding videos. Would JCodec allow me to fulfill my task? If yes, do you know any examples.

3. Compiling FFMPEG for Android would also be an option. However, I think it is a bit overkill to write FrameGrabber and FrameRecorder my self. I think that there must exist some solution besides the one of JavaCV.

4. Starting with API 18 there are the MediaCodec and the MediaMuxer in Android. Perhaps they can help me?

So lets come to my requirements. I'm currently targeting Android API 19, so I have every function available which I need. The most important requirement for me is the following:

If I process a video of 10 seconds with 30 FPS, the result should also be a video of 10 seconds with 30 FPS. So I want an exact copy of the video but with some drawing added to each frame by OpenCV. Using OpenCV via Python for example can do this task by using the VideoWriter class and VideoInput class. I need the same functionality on Android.

I am wondering that no one had this problem so far (or I did not find it).

Hopefully, I explained everything.

Answer 1

@Alekz: Yes, I found a solution. Even if it is additional overhead, it was sufficient for my research project.

The solution covers the usage of OpenCV 4 Android and JavaCV. I first use JavaCV to read in a frame one by another. Afterwards, I convert this frame to the format used by OpenCV, process the frame. Finally, I either display the processed frame directly on the screen or I convert it back to JavaCV and store it in a new file. Below you find the detailed solution for the different used frame formats.

1. Grab three channel IplImage frame from video with the FrameGrabber
2. Convert grabbed three channel frame to four channel IplImage frame
3. Convert four channel frame to Android Bitmap image
4. Convert Android Bitmap image to OpenCV Mat frame
5. Process OpenCV Mat frame
6. Convert Processed Mat frame to Android Bitmap image
7. Convert Android Bitmap image to four channel IplImage frame
8. Convert four channel IplImage frame to three channel IplImage
9. Store three channel IplImage as frame in the video with the FrameRecorder