60/120fps preview data on an Android device?

Question

We would like to process 120 fps (or 60fps) camera preview data using Neon assembly or GPU shaders to perform a simple image processing operation (eg 3x3 convolution). We are NOT interested in displaying the image on the screen so the camera data should go directly to a SurfaceTexture.

We thought we could do this using Camera2 API on a Nexus 5 using CameraConstrainedHighSpeedCaptureSession but have been unsuccessful.

There is no currently functional code on the internet (at least the part we have scraped) that allows this and yet it seems probable that the functionality exists.

The best example of using the Camera2 API for highspeed recording is here: https://github.com/PkmX/lcamera

In order to try to encourage someone to post a solution I am putting a bounty on this question.

Answer 1

This is accomplished in pjsip. Pull the repo to your machine:

svn checkout -r 4687 https://svn.pjsip.org/repos/pjproject/trunk

Then open the pjmedia/src/swig folder for an example app that renders video data incoming from the wire directly to a SurfaceTexture.

I have uploaded the two most relevant C source files (at a glance—there are certainly others) for you, here:

1. android_dev.c
2. android_opengl.c

Naturally, those files contain lots of irrelevant code for you, but hopefully have the snippet you're looking for.

You may need to first follow some of the instructions found here in order to generate the pjmedia/src/pjmedia-videodev/android/PjCamera*.java files that handle the minimalist Java/C interfacing if you are unsure how to do this. I was able to follow their instructions—using exactly the same versions for the NDK (works w/10d or 10e) and openssl (1.0.2a)—and successfully render video and create a custom app. This source accomplishes exactly what you're looking for if you just modify the C-level video capture to use your API via JNI hooks the same way they do.

For an example of using JNI hooks to change the API, have a look at their audiodev folder in pjmedia.

If you do build their project, be sure to follow the instructions at the very bottom for the newer NDK versions for building openssl 1.0.2a. Note the capital 'C' on ./Configure, as executing the standard ./configure may launch the wrong configuration process. I suggest copy and pasting their provided commands into a script. You may be able to skip openssl entirely for your purposes; I was only able to get my Android devices registering SIP accounts consistently once I integrated the openssl support.

Answer 2

We succesfully modified the camera2video sample to obtain 120 fps data rate on a Samsung S6.

The code is here.

We have not yet managed to extract the pixels but will update here when we do.