How to automatically rotate video based on camera orientation while recording?

Question

I am developing Mediasoup SFU and client web app, on the server I am recording client's stream by sending it to FFmpeg as plain RTP. FFmpeg creates HLS recording (.m3u8 and .ts files), because I need to be able to switch between WebRTC live stream and HLS recording before live stream ends.

My problem is that, when I am testing the app and I rotate my phone 90 degrees, the recording's aspect ratio stays the same but the image is rotated (as shown on images 1.1, 1.2 and 1.3 below). I need for it to change aspect ratio dynamically according to camera orientation. How can I do that using FFmpeg?

On the live stream it works perfectly fine (as shown on the images - 2.1 and 2.2 below), when the phone is rotated, the aspect ration is changed and video is shown correctly. I think it works on live stream because maybe WebRTC is signaling orientation changes somehow (but it does not project to the recording).

These are my ffmpeg command arguments for recording (version 6.1.1-3ubuntu5):

let commandArgs = [
      "-loglevel", "info",
      "-protocol_whitelist", "pipe,udp,rtp",
      "-fflags", "+genpts+discardcorrupt",
      "-reinit_filter", "1",
      "-strict", "-2",
      "-f", "sdp",
      "-i", "pipe:0",
      "-map", "0:v:0",
      "-c:v", "libx264",
      "-b:v", "1500k",
      "-profile:v", "high",
      "-level:v", "4.1",
      "-pix_fmt", "yuv420p",
      "-g", "30",
      "-map", "0:a:0",
      "-c:a", "aac",
      "-b:a", "128k",
      "-movflags", "+frag_keyframe+empty_moov",
      "-f", "hls",
      "-hls_time", "4",
      "-hls_list_size", "0",
      "-hls_flags", "split_by_time",
      `${filePath}.m3u8`
    ];

• Image 1.1 - Portrait mode in recording:

  

• Image 1.2 - Landscape mode in recording (rotated 90deg to my left side - front camera is on my left side):

  

• Image 1.3 - Landscape mode in recording (rotated 90deg to my right side):

  

• Image 2.1 - Portrait mode in live stream (correct behavior):

  

• Image 2.2 - Landscape mode in live stream (correct behavior):

  

Answer 1

The mediasoup documentation has a "tricks" page which seems to shed some light on what's happening.

There exist RTP capabilities and one of them is urn:3gpp:video-orientation which, (if supported) will use RTP header extensions to signal video orientation. This is apparently not supported by FFmpeg (or Firefox) and might be the cause of the issue you're facing.

By filtering the capabilities to remove that capability, it seems that it would cause the device to actually change the rotation of the video stream instead. The tricks page has the following code snippet to implement this:

// Let's get router RTP capabilities via our own app signaling.
let routerRtpCapabilities = await mySignaling.request("getRouterRtpCapabilities");

// Just for Chrome, Safari or any libwebrtc based browser.
if (supportsVideoOrientationHeaderExtension)
{
  // Remove the "urn:3gpp:video-orientation" extension so when rotating the
  // device, Chrome will encode rotated video instead of indicating the video
  // orientation in an RTP header extension.
  routerRtpCapabilities.headerExtensions = routerRtpCapabilities.headerExtensions.
    filter((ext) => ext.uri !== 'urn:3gpp:video-orientation');
}

// Finally apply the router RTP capabilities to the device.
await device.load({ routerRtpCapabilities });