Correctly orient raw camera data

Question

My camera app processes raw camera frames from onPreviewFrame() with OpenCV and then displays them on the screen. However, the raw frames are oriented the way the camera is mounted on the phone, which is usually not right-side up. To fix this, I rotate them manually with OpenCV, which is time-consuming.

I've looked into using setDisplayOrientation, but the documentation states that

This does not affect the order of byte array passed in onPreviewFrame

and I need the data to actually be right-side up, not just display right-side up. How can I correctly orient the raw camera data? If this is impossible, can I efficiently rotate the byte array passed to me in onPreviewFrame(), say by using OpenGL?

Answer 1

No, there's no way to get the API to do this for you, but to figure out how you need to rotate the data, take a look at the sample code at Camera.Parameters.setRotation. While setRotation() only affects JPEGs, you want to apply the exact same amount of rotation to the data you get in onPreviewFrame() as you would to the JPEGs.

Reproducing the sample code here, with minor changes:

Click to copy

public void onOrientationChanged(int orientation) {
   if (orientation == ORIENTATION_UNKNOWN) return;
   android.hardware.Camera.CameraInfo info =
        new android.hardware.Camera.CameraInfo();
   android.hardware.Camera.getCameraInfo(cameraId, info);
   orientation = (orientation + 45) / 90 * 90;
   int rotation = 0;
   if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
      rotation = (info.orientation - orientation + 360) % 360;
   } else {  // back-facing camera
      rotation = (info.orientation + orientation) % 360;
   }
   mCameraOrientation = rotation; // Store rotation for later use
}

...

void onPreviewFrame(byte[] data, Camera camera) {
   switch(mCameraOrientation) {
      case 0:
          // data is correctly rotated
          break;
      case 90:
          // rotate data by 90 degrees clockwise
      case 180:
          // rotate data upside down
      case 270:
          // rotate data by 90 degrees counterclockwise
   }
 }

So you need to inherit from OrientationEventListener and override onOrientationChanged as above, and then use the calculated orientation value from there to rotate the preview frames when they come in.

Answer 2

setDisplayOrientation only rotates the camera vs. surface you are drawing into, if I recall correctly. I believe you want to rotate the camera interpretation, what you may could achieve with setRotation