Using a SurfaceTexture filled by Camera preview as Renderscript input Allocation in Jelly Bean

Question

In Jelly Bean, is it possible to create a Renderscript allocation out of a SurfaceTexture filled by the camera preview ? I am building my application from within the Android source tree so I'm ok using @hide APIs such as Allocation.setSurfaceTexture(). However I would like to avoid using RS Graphics deprecated APIs. The similar question here did not get fully answered and was not JB-specific.

I have the following issues when trying the code below:

• The data getting into Renderscript is always zero
• For the onFrameAvailable callback to get repeatedly called, I have to updateTexImage() since when I call Allocation.ioReceive(), it does not get called back anymore after the first time, and there's an "invalid EGLDisplay" in the logcat. Yet I thought ioReceive() is the way to go - it internally also updateTexImage().
• Supported Allocation types include RGBA8888 but not NV21 (which is the camera preview format), how will the RS code be able to address data formatted this way ?

(I know the device I'm working with does support the requested VGA resolution).

public class SampleRSCPCActivity extends Activity implements SurfaceTexture.OnFrameAvailableListener {
final static int DO_KERNEL = 0;
private static final String TAG="SAMPLERSCP";
private static Camera mCamera;
private Camera.Parameters mParams;
private int mFrameWidth, mFrameHeight;
private static SurfaceTexture mST;
private RenderScript mRS;
private Allocation mInAllocation;
private Allocation mOutAllocation;
private ScriptC_mono mScript;

public void onCreate(Bundle savedInstanceState) {

    super.onCreate(savedInstanceState);

    Log.i(TAG, "onCreate()");
    createGUI();
    createCamera();
    createRSEnvironment();
}

public void onPause() {
    Log.i(TAG, "onPause");
    mCamera.stopPreview();
    mCamera.release();
    mCamera = null;
    super.onPause();
}

private void createCamera() {
    mCamera = Camera.open();
    mParams = mCamera.getParameters();

    mFrameWidth = 640;
    mFrameHeight = 480;
    mParams.setPreviewSize(mFrameWidth, mFrameHeight);
    mParams.setPreviewFormat(ImageFormat.NV21);

    mCamera.setParameters(mParams);
}

private void createRSEnvironment () {
    mRS = RenderScript.create(this);
    mScript = new ScriptC_mono(mRS, getResources(), R.raw.mono);

    Type.Builder b = new Type.Builder(mRS, Element.U8(mRS));

    int usage = Allocation.USAGE_SCRIPT | Allocation.USAGE_IO_INPUT;
    mInAllocation = Allocation.createTyped(mRS, b.setX(mFrameWidth).setY(mFrameHeight).create(), usage);
    mOutAllocation = Allocation.createTyped(mRS, b.setX(mFrameWidth).setY(mFrameHeight).create());

    Log.i(TAG, "Getting SurfaceTexture from input Allocation");
    mST = mInAllocation.getSurfaceTexture();

    mST.setOnFrameAvailableListener(this);

    try {
      Log.i(TAG, "Setting SurfaceTexture for camera preview");
      mCamera.setPreviewTexture(mST);

      Log.i(TAG, "Starting preview");
      mCamera.startPreview();
      } catch (IOException e) {
      Log.e(TAG, "Oops, something got wrong with setting the camera preview texture");
    }
}

private void createGUI() {
    requestWindowFeature(Window.FEATURE_NO_TITLE);
    getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, 
            WindowManager.LayoutParams.FLAG_FULLSCREEN);

    setContentView(R.layout.main);
}

private Handler handler = new Handler() {
    @Override
    public void handleMessage(Message msg) {
        if (msg.what == DO_KERNEL)
          Log.i(TAG, "Calling RS kernel");
          mST.updateTexImage();
          // mInAllocation.ioReceive();
          mScript.forEach_root(mInAllocation, mOutAllocation);

          Log.i(TAG, "Finishing RS");
          mRS.finish();
          Log.i(TAG, "Ok");
    }
};

public void onFrameAvailable(SurfaceTexture st) {
  Log.i(TAG, "onFrameAvailable callback");
  handler.sendEmptyMessage(DO_KERNEL);
}

}

The RS code is fairly simple, just trying to detect non-null data:

void root(const uchar *v_in, uchar *v_out, uint32_t x, uint32_t y) {

if (*v_in != 0)
  rsDebug("input data non null !", *v_in);
*v_out = (x / 640) * 255;

}

Answer 1

Following-up on my own question:

Turns out that reading NV21 buffers from a SurfaceTexture filled by the camera is not possible. I had to modify the Android source code to experiment with this (for the experts: get the current buffer of the SurfaceTexture, then lock it to obtain a real buffer pointer - I did this in the RS Driver's rsdAllocationIoReceive()). This would have been great to avoid performing a buffer copy from camera to RS.

The latest JB (MR1 version) has a test application called LivePreview which performs RS processing on the camera preview. However, it uses application-allocated preview callback buffers, which are then copied to the input Allocation. It interestingly uses the new ScriptIntrinsicRgbToYuv class for the color conversion. The bulk of the conversion is hand-coded Neon assembly so presumably quite fast.

It might even be that the Nexus 10 (which has a Mali-backed RS Driver) does this on the GPU, I'd love to get my hands on this device to play with. (Donators welcome ;-) !)