What is the structure of a video stream?

Question

The end goal is to process the RGB data of a video.

I am trying to read the bytes of a file that I have created using ffmpeg.

ffmpeg -video_size 100x100 -framerate 20 -f x11grab -i :0.0 \
    -c:v rawvideo -pix_fmt rgb24 -video.nut

I wrote a node script to help make it easier to read the binary data if you need it. The output of my current file is:

Hex Binary   Row
47  01000111 0
40  01000000 1
11  00010001 2
10  00010000 3
03  00000011 4
00  00000000 5
00  00000000 6
00  00000000 7
68  01101000 8 

I see the spec for a .nut, but I can't figure it out. I would like to be able to parse out the RGB data for each frame so that I am left with a RGB matrix for each "image" in the video stream. Thanks!

Answer 1

Personally, I like @Mulvya's answer. The .rgb format is much easier. However if you ever pass that file around, you will have to always include notes with it (such as : expected width, height, framerate etc), otherwise it's a sea of bytes with no idea where to stop per frame.

As for the .nut format as you previously asked...

Each video frame will be classed as a keyframe (since it's an uncompressed full image).

• First find the data section for your keyframes... Look for a start code sequence like this :
  4E 4B E4 AD EE CA 45 69.

• To be sure this is the keyframes data section, the next following 8 bytes are always set as :
  06 00 00 00 00 00 00 03.

• Then the next following bytes contain Flags and total bytes used for this keyframe (ie: 30000 for a 100 x 100 x 3 image). It gets tricky cos now you have to check at bit-level not just at byte-level... The short version of this story is (for a 100 x 100 image) is to just skip the next 4 bytes which should be 00 81 EA 30 to get to the 30 000 bytes of RGB data. et as :
  

  (a) First frame skip 4 bytes (should be 00 81 EA 30) + extract the following 30 000 bytes.

  (b) This leads to another keyframe code: 4E 4B E4 AD EE CA 45 69. Followed by 15 skip-able bytes (of which the last one is 30) and you get the 30K of image's RGB data.

  (c) For all other frames repeat step (b), whereby you : Skip 8 bytes of start code + Skip next following 15 bytes + Extract next 30 000 bytes for image. Repeat until end.

  • PS: As a final note... Those 4 bytes are only 4 because of total bits required for defining (flags etc) a 100 x 100 image. More bits will be used for a larger size image. In such a case, you really to parse the individual bits, also the final bits before a frame's data always give the size of bytes required for image extraction. Let me know if you need that info.

OLD ANSWER

I can't figure out what format the payload is in. I checked wikipedia for uncompressed video format, but that was no help...

-f mpegts means force the output format to be mpegts regardless of file extension.
So you really have MPEG TS format and it's never been RAW format. Those 3 letters, r-a-w, in your filename are misleading.

I can't be sure what format you actually need when you only say "uncompressed video". Is it RGB that you want? I only know of AVI and FLV as formats that support RGB frames (likely MOV can do it also but never tried it). In anycase you need a container for your RGB frame data

AVI container :

ffmpeg -video_size 1920x1080 -framerate 30 -f x11grab -i :0.0 -c:v rawvideo -pix_fmt rgb24 video.avi

FLV container :

ffmpeg -video_size 1920x1080 -framerate 30 -f x11grab -i :0.0 -c:v flashsv -pix_fmt rgb24 video.flv

PS : Maybe the info in this answer can help you decide on your output format & container.