Live Streaming ( Socket ) - How to sync audio and video?

Question

1- Which one is better to use for streaming video ? TCP or UDP socket and why?

2- While streaming live, audio and video are coming from the server separately, so how can i make sure that the video i display and the audio I play on the device are in sync?

Answer 1

I wrote a voice chat application a while ago and TCP was out of the question, UDP multicasting is really the only way to go if you're looking for near-realtime data flow. There's two main issues with streaming stuff over UDP though:

1. The dropped packets. In the case of audio, it's a pretty easy fix. Usually the dropped packets won't make an audible difference (the packets are decompressed individually). However, when dealing with video, especially if the video is compressed (it usually is), figuring out a proper transfer protocol that ensures network robustness is a daunting task to say the least, especially if you're doing this from scratch. Video frames are split up in various packets. Figuring out what to do when these packets are missing is tough.
2. Synchronization between audio and video. This is a very tough problem and I suggest reading up on protocols such as RTSP (Real-Time Streaming Protocol). This is not an easy task, but here's some introductory info: http://www.cs.columbia.edu/~hgs/rtsp/ - sometimes it's done by sending separate sync packets (some protocols send these over TCP) that tell the player how the sound should match up with the video.

Answer 2

I would do UDP. However it depends on what you want. UDP will drop packets rather than wait (TCP). The trade off is whether you want a stable, but sometimes slow and costly, or one that is efficient, but sometimes may not get delivered. The choice is yours when it comes to how you want to implement it and how you are using it.