Android Camera2 RAW streaming

Question

I am very new to Android and trying to:

• Stream raw data from the camera (ImageFormat RAW_SENSOR)
• Process raw data
• Display processed results interactively

It seems like raw data capture is only available in still capture mode. Is this correct ?

If so would it be possible to perform repeated CaptureRequests for RAW images instead ? What kind of performance can be expected using a 13MP sensor ? Any reference code ?

Many thanks in advance,

Guillaume

EDIT:

Here is what I have done so far:

• Create preview capture that renders to SurfaceTexture

• Frame rate is calculated / updated in onSurfaceTextureUpdated()

  Questions:

  • Is it OK to calculate framerate in onSurfaceTextureUpdated() ?
  • I tried to set different dimensions to texture.setDefaultBufferSize() but it has no effect on framerate. Is this normal ?

• Add raw ImageReader's surface to cameraDevice.createCaptureSession()

• Add callback to cameraCaptureSessions.setRepeatingRequest()
• In callback's onCaptureCompleted(), added a captureRawImage() call every N frames.
• captureRawImage() creates a still image capture.
• add setOnImageAvailableListener() to raw ImageReader to do something useful with raw image.

The raw resolution is 4208x3120

Minimum frame duration is 33ms

Stall duration is 200ms

The framerate I get for only preview is about 25-30 fps.

The framerate I get when I preview and enable raw capture every frame is about 15 fps.

I get some "ImageReader_JNI: Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers" message, even if I do raw capture every 100 frames.

I would love to get some feedback on:

• Whether this is a proper way to do what I want to do.
• How I could switch to burst capture rather than still image every N frame, if it would be more efficient.
• Whether these initial numbers make sense.

Many thanks,

Guillaume

Answer 1

Whether RAW capture is supported at all, and what rate it can be done are both device-dependent.

If the device supports the RAW capability, then you can use an ImageReader with the RAW_SENSOR format as a capture target. The format will then also be listed in the available output formats. You can see what kinds of stream combinations are supported in the documentation for createCaptureSession in the RAW-capability table.

To determine how fast you can capture RAW buffers, take a look at the output of getOutputMinFrameDuration, which tells you what the maximum frame rate when including an output of given format and resolution combination. In addition, the getOutputStallDuration method will tell you if you need extra delay between two consecutive requests targeting the RAW output to avoid stalling preview. There's no requirement to support any particular output rate of RAW buffers in the general case.

However, if the device supports either of the more-efficient RAW10 or RAW12 formats, and it supports the BURST_CAPTURE capability, then it's guaranteed to support capturing those formats with at least 20fps rates.

You can look at the Google camera2raw sample for basic still capture. However, you can certainly capture RAW buffers continually instead of alongside JPEG buffers. But your frame rate may be very low in that case, and to preserve your preview frame rate, you may wish to only issue a RAW capture one out of every N preview frames.

For example, if the stall duration for RAW_SENSOR is 100ms, but the minimum frame duration is 30ms, you can run continuous preview by having a capture request pattern of (R+P),P,P,P,(R+P),P,P,P... so that there's at least 100 ms between the starts of each raw capture. That'll get you ~8fps RAW and 30fps preview.