Defragment H264 NAL stream (Originally 1722 avb packets)

Question

Task at Hand :

Capture 1722 avb video packets coming through ethernet port and play them as live video in android. The video packets are of NAL H.264 stream.

What is already available :

The code to read the data from Ethernet port and capture the packets is ready. So in short, I have the payload data with me.

What I am looking for :

1. C code that can analyze these NAL H264 packets.
2. Identify the start, intermediate and end frames from the continuous stream of payloads .
3. Combine all the related H264 NAL payloads to form a video frame.

I guess the above process is called de-fragmentation. Once de-fragmented, I will then send this video frame to android video view and display them on the screen.

Any helpful resources will be really appreciated.

Answer 1

First, I assume your analysis is to be carried on NAL units. Below table shows the partial list of NAL unit types. In major H264 encoder implementations, you may find NAL unit types 1, 5, 6, 7, and 8 only. You may find other NAL units very rarely.

Access Unit Delimiter:

Your issue is simpler to solve if the stream has NAL unit number 9 i.e., Access Unit delimiter. All the NAL units, between 2 Access Unit delmiter NAL units, belong to single Video frame. Since this NAL unit type is a optional one, most encoder generally skip embedding this NAL unit. So. it is highly probable that you may not find this NAL unit in your stream

NAL Units - 6 & 7:

These 3 NAL units don't directly participate in de-framentation, but they are needed to for decode operation. In most cases, these 2 types come only once in a sequence i.e., at the beginning of video sequence.

NAL Units - 1 & 5:

These are the NAL units that are critical for de-fragmentation. For a given video frame, all NAL units should carry same NAL unit i.e., either 1 or 5. These NALs carry the slices of the frame. I assume slices come-in-order as ASO (Arbitrary slice order) support is extremely rare feature to find in encoders. The first slice of the frame carries a flag to indicate that it is the start of a video frame.

Above picture is formed by combining 2 partial tables (only that are relevant here) from H264 standard.

Once you decode NAL header (1-byte information), you will figure out whether it is of NAL type 1 or 5 (slice NAL unit). Once NAL is found as a slice unit, parse the stream for "first_mb_in_slice" symbol (this info comes immediately after 1-byte of NAL header info). If this flag is set, then this is the first slice of the video frame. The following NAL units will have this flag as zero till last slice of the current video frame. If a slice NAL unit's "first_mb_in_slice" flag is found to be set, that means this new slice belongs to next video frame and is the start of next video frame.

I hope these details will help in resolving your issue