Streaming multi-track audio using DASH

Question

We are creating a web browser based music streaming service which streams files containing separate, synchronzied audio tracks. The files are basically mp4s, but with some additional boxes.

Streaming will be done following the MPEG-DASH standard, and hopefully without reinventing the wheel (for example, there's dash.js).

---

Update 160525 - Adds some more background:

MPEG-DASH is a requirement for the project, both for its features (adaptiveness, easy server setups etc.) and because the project I'm working on only uses MPEG standards.

The streaming itself is on-demand playback of static files, i.e. no live streaming. The playback will have two modes; a static mode which just plays the tracks, and an interactive mode where the user will be able to change volume, panning etc. For now, static playback is fine.

---

Now, streaming synced multi-track audio doesn't seem to be the most common thing out there, and I've run into a couple of problems:

1. Following the MPEG-DASH standard

MPEG-DASH seems to be to be highly focused on video streaming.

Are there any documents or guidelines regarding audio-only streaming that I'm not able to find?

2. HTML5 elements vs Web Audio API

AFAIK, the <video> and <audio> elements only supports one audio track at a time. They provide lots of magic via MediaSource.addSourceBuffer(), where on-the-fly decoding of incoming buffers is taken care of.

Two possible ways of getting around this is:

1. Use multiple <audio> elements and sync them manually
2. Use the Web Audio API to (a) combine the buffers into a single signal sent to the audio context's destination, or (b) create AudioNodes for each track and sync them manually. This means we have to take care of decoding

Are there any other or better ways to stream multiple, synchronized audio tracks?

3. Using existing libraries

The dash.js reference player follows the DASH-AVC/264 guidelines, which to my understanding addresses video streaming and restricts the number of simultaneously playback audio tracks to one.

It also uses the HTML5 <video> element, which leads to the issues in 2. above.

It does, however, contain huge amounts of DASH related features, such as adaptive streaming, MPD manifest parsing etc etc.

Is dash.js a good suit for multi-track music streaming, or will it end up being a hack?

---

I am more than happy to answer questions or be corrected if I have misunderstood anything!

Answer 1

Some answers that I know:

1. You are right that DASH discussion is mostly focused on video, but audio-only DASH is still very doable.
2. a) The Media Source Extensions API (MediaSource.addSourceBuffer()) is absolutely the right way to implement DASH playback of a single source in JavaScript. DASH & MSE were developed at the same time with the intent of working together. Look for more references around MSE and DASH.
3. b) Once you have a single DASH source stream playing via MSE, then you would need to work on mixing the two sources together into a single output and that is probably going to use the Web Audio API, although I am not as familiar with the details of that API.

I will add a comment that combining two tracks on the client side does not seem like a good idea unless there are requirements you didn't mention. It makes more sense to me to do the mix on the server side with ffmpeg or similar media server tools then your client would only have to worry about a single stream. If the different source streams need to be synchronized that is probably not realistic to pull off in JavaScript in browser.