Decoding raw h.264 using Broadway.js

Question

I'm working on an application which gives me an encoded h.264 frame using ffmpeg:

avcodec_encode_video2(c, &packet, frame, &got_output)

If I save all packet.data to a file out.h264, it shows the desired output using ffplay.

Now, my goal is to send every packet as I receive it and display it (live-stream) on a webpage. For this purpose I'm using Broadway.js

I can confirm that the data I sent from the application is received correctly in the browser. However, I'm unable to display the same on a webGL canvas using Broadway (Player.js, Decoder.js, YUVCanvas.js) :

if (data != null) player.decode(new Uint8Array(data));

The output I get is a blank white canvas. data is an ArrayBuffer which contains the h.264 bitstream from the packet received from avcodec_encode_video2. Am I doing something wrong? Is the data supposed to be in a specific format?

side note:
My video file out.h264 plays properly using the sample provided here: BroadwayStream

It seems to use the command line ffmpeg interface and processes every packet as is received. My program using the ffmpeg libraries to get the same packets, which I need to render. Can someone help?

Answer 1

Keep in mind that Broadway only handles Baseline Profile encoded with CAVLC (Huffman coding), and just ignores Main Profile, or anything encoded with CABAC (Arithmetic coding). In other words, Broadway only accepts the easiest-to-decode type of H.264 stream. It can be frustrating when your video is coded incorrectly, because you get the dreaded blank screen syndrome.

BroadwayStream uses ffmpeg on the server side to debox mp4 files, converting them into raw H.264 data streams. That is, it generates a sequence of H.264 Network Access Layer Units (NALUs).

Then the browser side slurps that whole data stream and passes it to Broadway's decode() method. So, decode() always gets a sequence of complete NALUs, with no partial NALUs passed in any particular method invocation.

decode() doesn't take partial NALUs. (BroadwayStream neatly sidesteps this by giving it the whole stream.) And it doesn't take either mp4 or webm data streams, only the deboxed H.264. If you pass it partial NALUs it doesn't work--it just ignores the data you gave it. So your client-side js that accepts a datastream must, somehow, separate the stream into NALUs before passing them to decode().

If your js accepts muxed files of the MIME type video/webm; codecs="avc1.42E01E" you can look at https://github.com/themasch/node-ebml and this gist for ways to debox them. Each data chunk contains one or more whole NALUs which you can pass to decode().

If it's video/mp4; codecs="avc1.42E01E" fragmented MP4, you can debox the sps and pps items in the avcC box, and then the mdat boxes. Each of those items is one or more whole NALUs.

If you're getting a stream of raw H.264 from your server, you have to, somehow, parse it yourself into NALUs. Yumi Chan wrote a helpful article about NALUs. Keep in mind there are two ways to separate NALUs: packet transfer and byte stream. decode() accepts either, but your parser must too.