Android - Rotate video frames before sending to Wowza Streaming Engine using WebRTC

Question

I want to stream video from android camera to Wowza Streaming Engine (WSE) using WebRTC. When device in landscape mode, everything work well. Then I try to stream by putting the device in portrait mode.

The first thing I notice in WSE player is the video stream has been rotated 90 counter-clockwise. I figured out that WebRTC do not rotate each video frame that comes out from onPreviewFrame API before sending to WSE and unfortunately WSE do not support any mechanisms to rotate video frame at their side at least so far.

So I checked out WebRTC android native source code and modify it to rotate each video frame before sending to WSE. And right now I can see the video stream in portrait mode in WSE player.

But it has an issue is sometimes video stream look so weird. Please see the following images.

A normal image

A weird image

I put camera to a fix position. At the first time WSE player displays the first one, but sometimes the second one is showed up.

And here is the file in WebRTC source code that I changed. ~/webrtc/src/sdk/android/src/jni/androidvideotracksource.cc

void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
                                                        int length,
                                                        int width,
                                                        int height,
                                                        VideoRotation rotation,
                                                        int64_t timestamp_ns) {
  RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());

  int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
  int64_t translated_camera_time_us =
      timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());

  int adapted_width;
  int adapted_height;
  int crop_width;
  int crop_height;
  int crop_x;
  int crop_y;

  if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
                  &adapted_height, &crop_width, &crop_height, &crop_x,
                  &crop_y)) {
    return;
  }

  const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
  const uint8_t* uv_plane = y_plane + width * height;
  const int uv_width = (width + 1) / 2;

  RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));

  // Can only crop at even pixels.
  crop_x &= ~1;
  crop_y &= ~1;
  // Crop just by modifying pointers.
  y_plane += width * crop_y + crop_x;
  uv_plane += uv_width * crop_y + crop_x;

  rtc::scoped_refptr<I420Buffer> buffer =
      buffer_pool_.CreateBuffer(adapted_width, adapted_height);

  nv12toi420_scaler_.NV12ToI420Scale(
      y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
      buffer->MutableDataY(), buffer->StrideY(),
      // Swap U and V, since we have NV21, not NV12.
      buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
      buffer->StrideU(), buffer->width(), buffer->height());

  // TODO: Rotate I420 frame 90 degrees clockwise.
  rtc::scoped_refptr<I420Buffer> rotated_buffer =
      I420Buffer::Rotate(*buffer, kVideoRotation_90);

  OnFrame(VideoFrame(rotated_buffer, rotation, translated_camera_time_us));
}

I added this line of code to rotate I420 frame 90 degrees clockwise.

// TODO: Rotate I420 frame 90 degrees clockwise.
  rtc::scoped_refptr<I420Buffer> rotated_buffer =
      I420Buffer::Rotate(*buffer, kVideoRotation_90);

I would appreciate any help!

Answer 1

Finally I have figured out a solution to solve this problem. Below are my steps:

Step 1: Make sure you lock your streaming activity to portrait orientation

Step 2: Change this method in the file in WebRTC android native source code ~/webrtc/src/sdk/android/src/jni/androidvideotracksource.cc

Original Version:

void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
                                                        int length,
                                                        int width,
                                                        int height,
                                                        VideoRotation rotation,
                                                        int64_t timestamp_ns) {
  RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());

  int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
  int64_t translated_camera_time_us =
      timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());

  int adapted_width;
  int adapted_height;
  int crop_width;
  int crop_height;
  int crop_x;
  int crop_y;

  if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
                  &adapted_height, &crop_width, &crop_height, &crop_x,
                  &crop_y)) {
    return;
  }

  const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
  const uint8_t* uv_plane = y_plane + width * height;
  const int uv_width = (width + 1) / 2;

  RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));

  // Can only crop at even pixels.
  crop_x &= ~1;
  crop_y &= ~1;
  // Crop just by modifying pointers.
  y_plane += width * crop_y + crop_x;
  uv_plane += uv_width * crop_y + crop_x;

  rtc::scoped_refptr<I420Buffer> buffer =
      buffer_pool_.CreateBuffer(adapted_width, adapted_height);

  nv12toi420_scaler_.NV12ToI420Scale(
      y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
      buffer->MutableDataY(), buffer->StrideY(),
      // Swap U and V, since we have NV21, not NV12.
      buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
      buffer->StrideU(), buffer->width(), buffer->height());

  OnFrame(VideoFrame(rotated_buffer, rotation, translated_camera_time_us));
}

Modified Version:

void AndroidVideoTrackSource::OnByteBufferFrameCaptured(const void* frame_data,
                                                        int length,
                                                        int width,
                                                        int height,
                                                        VideoRotation rotation,
                                                        int64_t timestamp_ns) {
  RTC_DCHECK(camera_thread_checker_.CalledOnValidThread());

  int64_t camera_time_us = timestamp_ns / rtc::kNumNanosecsPerMicrosec;
  int64_t translated_camera_time_us =
      timestamp_aligner_.TranslateTimestamp(camera_time_us, rtc::TimeMicros());

  int adapted_width;
  int adapted_height;
  int crop_width;
  int crop_height;
  int crop_x;
  int crop_y;

  if (!AdaptFrame(width, height, camera_time_us, &adapted_width,
                  &adapted_height, &crop_width, &crop_height, &crop_x,
                  &crop_y)) {
    return;
  }

  const uint8_t* y_plane = static_cast<const uint8_t*>(frame_data);
  const uint8_t* uv_plane = y_plane + width * height;
  const int uv_width = (width + 1) / 2;

  RTC_CHECK_GE(length, width * height + 2 * uv_width * ((height + 1) / 2));

  // Can only crop at even pixels.
  crop_x &= ~1;
  crop_y &= ~1;
  // Crop just by modifying pointers.
  y_plane += width * crop_y + crop_x;
  uv_plane += uv_width * crop_y + crop_x;

  rtc::scoped_refptr<I420Buffer> buffer =
      buffer_pool_.CreateBuffer(adapted_width, adapted_height);

  nv12toi420_scaler_.NV12ToI420Scale(
      y_plane, width, uv_plane, uv_width * 2, crop_width, crop_height,
      buffer->MutableDataY(), buffer->StrideY(),
      // Swap U and V, since we have NV21, not NV12.
      buffer->MutableDataV(), buffer->StrideV(), buffer->MutableDataU(),
      buffer->StrideU(), buffer->width(), buffer->height());

  // TODO: Comment out this line of code to apply custom code.
  // OnFrame(VideoFrame(buffer, rotation, translated_camera_time_us));

  // TODO: The custom code to rotate video frame before passing
  // to next layers of WebRTC.

  // Rotate I420 frame rotation degrees.
  // Value of the rotation is 90 or 270 based on camera orientation.
  rtc::scoped_refptr<I420Buffer> rotated_buffer =
      I420Buffer::Rotate(*buffer, rotation);

  // Make sure the I420 frame has valid side in portrait mode.
  rtc::scoped_refptr<I420Buffer> final_buffer =
      buffer_pool_.CreateBuffer(height, width);
  final_buffer->ScaleFrom(*rotated_buffer);

  // After rotating the I420 frame, set value of the rotation to 0.
  // This mean we do not want to rotate the frame in next layers anymore.
  rotation = kVideoRotation_0;

  // Pass processed frame to the next layers.
  OnFrame(VideoFrame(final_buffer, rotation, translated_camera_time_us));
}

Now my stream is displayed perfectly on Streaming Wowza Engine Player.