FFmpeg auto rotates video when only copying stream

Question

I am facing this issue while converting mp4 (portrait) file to mkv. The command I'm using

ffmpeg -y -i test.mp4 -vcodec copy -acodec copy test.mkv

The output video is 90 degree counter clockwise rotated. Its because I think the side data is being removed.

Side data: displaymatrix: rotation of -90.00 degrees

Input file test.mp4 info

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2019-02-23T11:18:50.000000Z
    com.android.version: 8.0.0
  Duration: 00:00:25.86, start: 0.000000, bitrate: 12270 kb/s
    Stream #0:0(eng): Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720, 12005 kb/s, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      rotate          : 90
      creation_time   : 2019-02-23T11:18:50.000000Z
      handler_name    : VideoHandle
    Side data:
      displaymatrix: rotation of -90.00 degrees
    Stream #0:1(eng): Audio: aac (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 256 kb/s (default)
    Metadata:
      creation_time   : 2019-02-23T11:18:50.000000Z
      handler_name    : SoundHandle

Rotated output file test.mkv info

Input #0, matroska,webm, from 'test.mkv':
  Metadata:
    MAJOR_BRAND     : mp42
    MINOR_VERSION   : 0
    COMPATIBLE_BRANDS: isommp42
    COM.ANDROID.VERSION: 8.0.0
    ENCODER         : Lavf58.12.100
  Duration: 00:00:25.87, start: 0.000000, bitrate: 12265 kb/s
    Stream #0:0(eng): Video: h264, yuv420p(tv, bt709, progressive), 1280x720, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 2k tbc (default)
    Metadata:
      ROTATE          : 90
      HANDLER_NAME    : VideoHandle
      DURATION        : 00:00:25.866000000
    Stream #0:1(eng): Audio: aac, 48000 Hz, stereo, fltp (default)
    Metadata:
      HANDLER_NAME    : SoundHandle
      DURATION        : 00:00:25.813000000

Converting the rotated mkv to mp4 again works fine and I get the portrait file. The displaymatrix side data appears again in the file info.

Also converting the same mp4 file to m4v by copying the stream works fine.

In this post they solved it for c++. I am working on android and using ffmpeg android wrapper to use the ffmpeg library. Is there any ffmpeg flag to handle this situation?

Answer 1

The reason why this works in MP4/MOV is because for this format the display matrices are stored in the Movie Header (mvhd) and Track Header (tkhd) atoms.

Example layout for mvhd showing the matrix structure:

Source: Apple QuickTime File Format Specification

When the stream is muxed in MKV the matrices are lost and the muxer falls back to adding the ROTATE metadata entry instead. You can see this in your ffprobe output. If the stream is then muxed back in a MP4 the metadata information is used to create the matrix (if you pass -map_metadata -1 when creating the MKV the information is lost and the conversion no longer works).

On the player side there's no guarantee that it will look for and apply the rotation specified in the ROTATE tag. If you want to use MKV with the proper orientation you will have to re-encode, otherwise stick with MP4.