How to play raw NAL units in Android exoplayer?

Question

I know that exoplayer has support for RTSP, but I need C++ code that works on players from lots of OSs, so I need to parse the RTP packet in C++ to NAL units before passing to exoplayer

I found a way to decode RTP packets using live555 and extract its NAL units. According to ExoPlayer's documentation:

Components common to all ExoPlayer implementations are:

A MediaSource that defines the media to be played, loads the media, and from which the loaded media can be read.

A MediaSource is injected via ExoPlayer.prepare at the start of playback. ...

So I need a custom MediaSource that can extract NAL units from my C++ code.

At the class reference for MediaSource we can see that there are already some MediaSources available. I though maybe SmoothStreaming MediaSource could work but there's no description of what it does exactly and in its constructor I have to provide an Uri or an SsManifest (what).

I can see that there exists a NAL unit utility in this library so maybe things are already half done

So how to build or use an already available MediaSource to read NAL units for ExoPlayer to play?

As an additinal, how would you pass the NAL units from C++ to Java? In the code I found it's simply storing them in a C++ buffer. Should I read this buffer in Java somehow?

UPDATE:

I've been researching how this library works. It all begins with a MediaSource which have objects like Extractor and DataSource. Seems that ExtractorMediaSource is a MediaSource where you can provide your own Extractor and DataSource.

As I understood, DataSource is a class that gets the raw bytes from any possible place, be it a file read or a network packet. Based on the available Extractor classes on the library like Mp4Extractor and Mp3Extractor, an Extractor is something that will interpret the data read from DataSource. The two main methods from the Extractor interface are:

void init(ExtractorOutput output)
int read(ExtractorInput input, PositionHolder seekPosition)

I don't know what are ExtractorInput and ExtractorInput for, but they look important.

So somehow Extractor reads from DataSource, parses it and sends to Renderer in a common format?

I need to know how is this common format so I can parse the NAL units that I read from a custom DataSource.

Answer 1

You cannot play NAL units or raw H.264 stream with ExoPlayer.
The picture data must exist within a supported container/format.

It's not clear what your mysterious C++ code is doing, is it an NDK setup? What's its role in Android decoding? Are you saying you're unable to pass [from the C++ function] a data array into some Android function as function parameter? Is it something like this Java to C++ setup? It's not clear what your real problem is...

If you insist on Exoplayer, I can tell you that FLV is the one (on containers list) that might be the best option since it can be built in real-time (re-muxing). You first create an FLV header that holds SPS and PPS data then followed by keyframe (extracted H264 data from the first NAL). You'll have to get familiar with FLV bytes structure, but each frame header is around 13 bytes followed by NAL data, repeat for each frame until end. This woud be realtime transcoding.

As a second option, for Android, you could just use MediaCodec to decode the H264 as extracted from the NAL units. Here is a useful example source. Just use as:

MediaCodec.createDecoderByType("video/avc"); //then later give NAL units

Study also functions of this other source for ideas of how it works to via Android's own decoder.

Other starting points:

• Android Decode raw h264 stream with MediaCodec (NAL units).

• Decode H264 raw stream using mediacodec