Decoding h264 ByteStream on Android

Question

I want to decode and display a raw h264 video byte stream in Android and therefore I'm currently using the MediaCodec/Format classes. I acquire the frame data via Udp from a server. But, unfortunately, nothing is displayed at the moment.

Here is what I have so far.

Initializing MediaCodec Class:

codec = MediaCodec.createDecoderByType("video/avc");

MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, "video/avc");
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 100000);
format.setInteger(MediaFormat.KEY_WIDTH, 800);
format.setInteger(MediaFormat.KEY_HEIGHT, 600);
format.setInteger("max-width", 800);
format.setInteger("max-height", 600);
format.setInteger("push-blank-buffers-on-shutdown", 1);

codec.configure(format, surface, null, 0);

Usage of the Decoder:

int inIndex = codec.dequeueInputBuffer(10000);
if(inIndex >= 0)
{
                ByteBuffer inputBuffer = codecInputBuffers[inIndex];
                inputBuffer.clear();
                inputBuffer.put(frameData);
                codec.queueInputBuffer(inIndex, 0, frameSize, 33, 0);
}

int outIndex = codec.dequeueOutputBuffer(null, 10000);

switch(outIndex)
{
  case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
    codecOutputBuffers = codec.getOutputBuffers();
    System.out.println("OB Changed");
    break;
  case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
        System.out.println("OF Changed");
        break;
  case MediaCodec.INFO_TRY_AGAIN_LATER:
    System.out.println("l8r");
    break;
  default:
    ByteBuffer buffer = codecOutputBuffers[outIndex];
    codec.releaseOutputBuffer(outIndex, true);
}

The Device I'm testing this code on is a Google Nexus 5. When I run this, the outIndex equals MediaCodec.INFO_TRY_AGAIN_LATER the whole time.

I've previously written a Client for Notebooks, which works fine, so I guess the h264 Stream from the Server should be OK.

Thanks for your help

Edit: In case someone is encountering the same problem, the proposed amendment (1) of fadden solved the issue. I corrected the code above. Its working now. Another mistake of my shown example was, that you cannot pass null to .dequeueOutputBuffers(...);. You have to do something like

        BufferInfo buffInfo = new MediaCodec.BufferInfo();
        int outIndex = codec.dequeueOutputBuffer(buffInfo, 10000);

even if you're not interessted in it. ;)

Answer 1

I see a few problems...

(1) You're attempting to replace a buffer in the input buffer array. MediaCodec doesn't work like this -- the framework provides the buffers, and you copy the data into them. The idea is that, by allowing the framework to do the allocation, it can potentially avoid copying the data later on.

You need to get the array of input buffers from decoder.getInputBuffers(), and use those. Make sure to clear() the ByteBuffer to reset the position and limit each time.

(2) You're writing a single packet of data and expecting a frame of output data. In practice, you may need to supply multiple buffers of data before the first frame is generated. See this post for an example. In some profiles the encoder is allowed to reorder frames, so even after the decoder starts going you can't just feed a frame and wait for decoded data to pop out the other side.

(3) The AVC decoder needs the SPS/PPS data, which you can provide via a buffer with the BUFFER_FLAG_CODEC_CONFIG flag set, or by adding the data with "csd-0" / "csd-1" keys to the MediaFormat using MediaFormat#setByteBuffer(). Examples of both approaches can be found in EncodeDecodeTest.

There are a number of AVC decoding examples on bigflake, but the data source is the MediaCodec encoder, so they generally get point #3 for free.

This posting may be useful for you.

For displaying the frames, you can see different approaches in Grafika (which generally works with .mp4 files, so the encode/decode implementation there isn't as relevant).