Decode mp4/h.264 using MediaCodec without MediaExtractor, expected access unit format

Question

I am trying to use the MediaCodec API for decoding without using the MediaExtractor API. Instead, i use mp4parser to get the samples from the mp4 files. For now, i am only using h.264 / avc coded video content.

The official documentation of the MediaCodec API states:

buffers do not start and end on arbitrary byte boundaries, this is not a stream of bytes, it's a stream of access units.

Meaning, i have to feed access units to the decoder. However, i miss some details in this information:

For h.264, in an mp4 sample, there can be multiple NAL units, that are each preceded by 4 (default) bytes specifying the NAL unit length.

Now my questions:

1. There can be mp4 samples, where codec config NAL units (sps, pps) are mixed with NAL units containing coded (parts of) frames. In that case, should i pass the flag BUFFER_FLAG_CODEC_CONFIG at the call of queueInputBuffers()?

2. There can also be other (additional) NAL units in mp4 samples, like SEI or access unit delimiter NAL units. What about those? No problem?

I tried different kinds of possibilities, but all the feedback i get from Android is that the calls of dequeueOutputBuffer() time out (or don't return, if i pass -1 as timeout parameter). As a result, i don't seem have a way to troubleshoot this issue.

Any advice what to do or where to look is of course very welcome as well.

Answer 1

The NAL length prefixes that specify the NAL unit length need to be converted to Annex-B startcodes (bytes 0x00, 0x00, 0x00, 0x01) before passing to MediaCodec for decoding. (Some decoders might actually accept the MP4 format straight away, but it's not too common.)

The SPS/PPS that is stored in the avcC atom in the file also needs to be converted to use Annex-B startcodes. Note that the avcC atom contains a few other fields that you don't need to pass on to the decoder. You can either pass the SPS and PPS packed in one buffer (with startcodes before each of them) with the BUFFER_FLAG_CODEC_CONFIG flag set before sending any actual frames, or pass them (with Annex-B startcodes) in the MediaFormat you use to configure the decoder (either in one ByteBuffer with the key "csd-0", or in two separate keys as "csd-0" and "csd-1").

If your file has got more SPS/PPS inside each frame, you should just be able to pass them as part of the frame, and most decoders should be able to cope with it (especially if it's the same SPS/PPS as before and not a configuration change).

Thus: Pass all NAL units belonging to one sample in one single buffer, but with all NAL unit length headers rewritten to startcodes. And to work with MP4 files that don't happen to have SPS/PPS inside the stream itself, parse the avcC atom (I don't know in which format mp4parser returns this) and pass the SPS and PPS with startcodes to the decoder (either via MediaFormat as "csd-0" or as the first buffer, with BUFFER_FLAG_CODEC_CONFIG set).