Video processing in Android

Question

I'm using Android 2.2 with Eclipse.

I would like to make an application that captures video, and for each frame, its send it as a bitmap to a method that processes it and returns a new bitmap and shows the processed bitmap.

I am not very familiar with Android, so please, can anyone send me to the resources I need to look at to do such a thing?

Answer 1

It is simple enough to accomplish the following steps using the Android SDK:

• Capture the preview frames from the camera as bitmapped data. Camera.PreviewCallback will return a byte[] of data representing the frame in a number of possible image formats.
• Modify the pixel data. Since the data comes back as raw bytes, making adjustments to the data is relatively easy...the difficulty here is applying an algorithm for the particular image processing you want to do. There aren't any built-in effects (pre-4.0) that can be applied simply to images so you will have to write your own.
• It is also possible to decode the data into a Bitmap object to make working with the pixels easier. In 2.2, you have the option of using the NDK and jnigraphics to work with a Bitmap's pixels in native code, which is significantly faster than at the Java layer.
• Take the contents of your resultant Bitmap and display it. For fast moving data, you would want to display this on a SurfaceView; using the lockCanvas() and unlockCanvasAndPost() methods available from the SurfaceHolder that view contains.

If this is all you are wanting to do, you could accomplish this with little difficulty. However, this is not the same as capturing video. Android does not currently provide hooks for you to stream frames out into an encoded video container (MPEG4, 3GP, etc.) in real-time. It's video capture capabilities are wrapped up tightly into the MediaRecorder which controls the process from frame capture all the way to writing the encoded video. You would need a 3rd part library such as FFMPEG (which has been built and run using the NDK layer a number of times in Android applications) to assist in the encoding process of your modified frames.

I know you have a 2.2 target, but Android 4.0 does provide some relief here as they have released a new version of the NDK which allows one to have more say in what happens when reading image data from a stream before handing it to the presentation layer. However, I have not spent enough time with it to know whether it could be recommended for your situation.