how to parse Access Unit in h.264

Question

I am working in a project that needs to cut some Access units in H.264 raw elementary stream,for example remove 4 access units and play the remaining video .

For this I took Access unit Delimiter (NAL Unit Type:9) as boundary for Access Unit and cut the video but the video ended with packet loss. But if I took Sequence parameter set (NAL Unit type :7) as boundary, resultant video playing without any packet loss.
Some one please help me how to solve this issue: where shall I cut the video ?

Answer 1

As you did not describe your stream a bit more in details, like what NALUs are present and how stuff is formatted here are some general guidelines:

You always need SPS/PPS as these contain information how to decode the frames. I assume you receive an Annex B kind of stream, in this case cache the last SPS/PPS until a new one comes along, decode remaining pictures with the old one and then apply the new one.

If you have AUD NALUs then you already have the end of the last access unit and you can split the video easily and not much needs to be done.

In case you need to dig deeper, here is an excerpt from specs regarding the order of the NAL units and how to detect the first VCL NAL unit of a primary coded picture.

From ITU-T H.264 (05/2013)

7.4.1.2.3 Order of NAL units and coded pictures and association to access units

This clause specifies the order of NAL units and coded pictures and association to access unit for coded video sequences that conform to one or more of the profiles specified in Annex A and are decoded using the decoding process specified in clauses 2-9.

An access unit consists of one primary coded picture, zero or more corresponding redundant coded pictures, and zero or more non-VCL NAL units. The association of VCL NAL units to primary or redundant coded pictures is described in clause 7.4.1.2.5.

The first access unit in the bitstream starts with the first NAL unit of the bitstream.

The first of any of the following NAL units after the last VCL NAL unit of a primary coded picture specifies the start of a new access unit:

• access unit delimiter NAL unit (when present),

• sequence parameter set NAL unit (when present),

• picture parameter set NAL unit (when present),

• SEI NAL unit (when present),

• NAL units with nal_unit_type in the range of 14 to 18, inclusive (when present),

• first VCL NAL unit of a primary coded picture (always present).

The constraints for the detection of the first VCL NAL unit of a primary coded picture are specified in subclause 7.4.1.2.4.

The following constraints shall be obeyed by the order of the coded pictures and non-VCL NAL units within an access unit:

• When an access unit delimiter NAL unit is present, it shall be the first NAL unit. There shall be at most one access unit delimiter NAL unit in any access unit.

• When any SEI NAL units are present, they shall precede the primary coded picture.

• When an SEI NAL unit containing a buffering period SEI message is present, the buffering period SEI message shall be the first SEI message payload of the first SEI NAL unit in the access unit.

• The primary coded picture shall precede the corresponding redundant coded pictures.

• When redundant coded pictures are present, they shall be ordered in ascending order of the value of redundant_pic_cnt.

• When a sequence parameter set extension NAL unit is present, it shall be the next NAL unit after a sequence parameter set NAL unit having the same value of seq_parameter_set_id as in the sequence parameter set extension NAL unit.

• When one or more coded slice of an auxiliary coded picture without partitioning NAL units is present, they shall follow the primary coded picture and all redundant coded pictures (if any).

• When an end of sequence NAL unit is present, it shall follow the primary coded picture and all redundant coded pictures (if any) and all coded slice of an auxiliary coded picture without partitioning NAL units (if any).

• When an end of stream NAL unit is present, it shall be the last NAL unit.

• NAL units having nal_unit_type equal to 0, 12, or in the range of 20 to 31, inclusive, shall not precede the first VCL NAL unit of the primary coded picture.

(NOTE 2 – Sequence parameter set NAL units or picture parameter set NAL units may be present in an access unit, but cannot follow the last VCL NAL unit of the primary coded picture within the access unit, as this condition would specify the start of a new access unit.)

(NOTE 3 – When a NAL unit having nal_unit_type equal to 7 or 8 is present in an access unit, it may or may not be referred to in the coded pictures of the access unit in which it is present, and may be referred to in coded pictures of subsequent access units.)

The structure of access units not containing any NAL units with nal_unit_type equal to 0, 7, 8, or in the range of 12 to 18, inclusive, or in the range of 20 to 31, inclusive, is shown in Figure 7-1.

7.4.1.2.4 Detection of the first VCL NAL unit of a primary coded picture

This clause specifies constraints on VCL NAL unit syntax that are sufficient to enable the detection of the first VCL NAL unit of each primary coded picture for coded video sequences that conform to one or more of the profiles specified in Annex A and are decoded using the decoding process specified in clauses 2-9.

Any coded slice NAL unit or coded slice data partition A NAL unit of the primary coded picture of the current access unit shall be different from any coded slice NAL unit or coded slice data partition A NAL unit of the primary coded picture of the previous access unit in one or more of the following ways:

• frame_num differs in value. The value of frame_num used to test this condition is the value of frame_num that appears in the syntax of the slice header, regardless of whether that value is inferred to have been equal to 0 for subsequent use in the decoding process due to the presence of memory_management_control_operation equal to 5.

(NOTE 1 – A consequence of the above statement is that a primary coded picture having frame_num equal to 1 cannot contain a memory_management_control_operation equal to 5 unless some other condition listed below is fulfilled for the next primary coded picture that follows after it (if any).

• pic_parameter_set_id differs in value.

• field_pic_flag differs in value.

• bottom_field_flag is present in both and differs in value.

• nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0.

• pic_order_cnt_type is equal to 0 for both and either pic_order_cnt_lsb differs in value, or delta_pic_order_cnt_bottom differs in value.

• pic_order_cnt_type is equal to 1 for both and either delta_pic_order_cnt[ 0 ] differs in value, or delta_pic_order_cnt[ 1 ] differs in value.

• IdrPicFlag differs in value.

• IdrPicFlag is equal to 1 for both and idr_pic_id differs in value.

(NOTE 2 – Some of the VCL NAL units in redundant coded pictures or some non-VCL NAL units (e.g., an access unit delimiter NAL unit) may also be used for the detection of the boundary between access units, and may therefore aid in the detection of the start of a new primary coded picture.)