Streaming audio and video in sync for mp4 container using Gstreamer framework

Question

I am new to gstreamer and I want to stream a mp4 video which is having audio and video both from my Host(Ubuntu PC) to target board. I have successfully streamed only video. But streaming both(audio and video) is not happening.

Pipeline to stream only video is as below

Sender:

gst-launch-1.0 -v filesrc location = video.mp4 ! decodebin ! x264enc ! rtph264pay ! udpsink host=192.168.2.136 port=5000

receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=3394826012, timestamp-offset=2215812541, seqnum-offset=46353" ! rtph264depay ! avdec_h264 ! videoconvert ! ximagesink sync=false

Hence, can see video on target.

Did streaming of only audio from mp4 file.

Pipeline to stream only video is as below

Sender:

gst-launch-1.0 -v filesrc location=video.mp4 ! qtdemux name=demuxer demuxer. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.2.69 port=5000

Receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=3386264266, timestamp-offset=2822202855, seqnum-offset=17719" ! rtpmp4adepay ! decodebin ! alsasink &

This too runs successfully.

But when I tried below pipeline to stream both audio/video in sync:

Sender:

gst-launch-1.0 -v filesrc location=sample.mp4 ! qtdemux name=demux demux. ! queue ! rtph264pay pt=96 ! udpsink host=192.168.3.101 port=5000 demux. ! queue ! rtpmp4apay pt=97 ! udpsink host=192.168.3.101 port=5001

Receiver:

gst-launch-1.0 -v udpsrc port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96, ssrc=2226494377, timestamp-offset=3242004369, seqnum-offset=17021" ! rtph264depay ! decodebin ! ximagesink udpsrc port=5001 caps="application/x-rtp, media=audio, clock-rate=44100, encoding-name=MP4A-LATM, cpresent=0, config=40002420, payload=97, ssrc=1499706502, timestamp-offset=190741668, seqnum-offset=24774" ! rtpmp4adepay ! decodebin ! alsasink

But: get an output as below:

ERROR: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin1/GstFaad:faad0: Could not decode stream.

fails. Please can someone suggest me a pipeline which can stream audio/video both on target device in sync.

Thanks in advance.

Answer 1

There seem to be issues with AAC in RTP as well as other RTP payloaders in gstreamer.

As mentioned in the other answer, it is anyway the best strategy to not demux and split video and audio if it is desired to have a synchronized playback on the receiver side. Additionally, a container format that is streamable can also improve overall functionality.

My proposal is therefore to remux the data streams into MPEG2-TS and transport that via RTP. This has several advantages, like good re-synchronization after packet loss and that A/V always stays sync.

Example

This works for me with gstreamer 1.2.4:

Sender

gst-launch-1.0 -v filesrc location=/foo/bar.mp4 ! \
  qtdemux name=demux demux. ! h264parse config-interval=10 ! queue ! mux. \
  mpegtsmux name=mux ! rtpmp2tpay ! udpsink host=10.0.0.1 port=5003 \
  demux. ! aacparse ! queue ! mux.

Receiver (file)

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse !  filesink location=/foo/bar.ts

Receiver (output)

This command is based on the commands in the question, could not test it with ALSA:

gst-launch-1.0 -v udpsrc port=5003 caps="application/x-rtp" ! \
  rtpmp2tdepay ! tsparse ! tsdemux name=demux \
  demux. ! queue ! audioconvert ! alsasink \
  demux. ! queue ! xvimagesink