HTML5: Playing live Opus audio frames without browser plug-in

Question

We need to replace our NPAPI browser plugin with a plugin-less solution. We have 3rd party input device that provides us live audio in the form of Opus 'frames'. We transmit those frames to the browser using binary WebSockets; and then, forward the data to our NPAPI plugin for decoding and audio playback. See picture.

What approach should we take to replace the NPAPI plugin with an HTML5-ish solution given these requirements?

1. Minimize end-to-end latency to no more than 3-5s (assumes 200mS round trip network latency).
2. Provide a means to apply audio filters (client/browser side)

Using the html5 audio tag seems to introduce a huge amount of latency as various browsers will require a certain amount of buffering (15-30s of audio) before beginning playback. We understand Opus may or may not be supported on all browsers. If needed (though we'd rather not to reduce bandwidth), we could encapsulate the Opus frames into a Ogg container within the web service before sending the data to the browser. Looking at one of the demos from html5rocks, HTML5 Audio Playground, it appears as though #2 is possible.

If this is a poor place to ask such a design question, please suggest other forums/groups that might be more appropriate.

Thanks for any help or suggests you might offer.

Answer 1

I have a similar situation. I have been using WebSockets and Media Source Extensions to play an MP3 feed in Google Chrome with little latency, but the MP3 codec is not supported by some other browsers when used with MSE. It turns out that most browsers, at least Chrome, Firefox, Opera and Edge, can play Opus natively with MSE, provided that it's properly encapsulated in an MP4 or WebM container.

Packing Opus in Ogg is quite simple, I converted some code I found from JavaScript to C#.

Packing Opus in WebM is somewhat more complicated. I wrote this C# code from scratch, based on the WebM/Matroska and EBML specs. When served via HTTP, it plays correctly in Chrome and Firefox, but VLC appears to be unable to stream Opus/WebM over HTTP. At least Chrome requires that the time sequence starts at 0, so packing on the server side was not a good option since that would require modifications to the distribution system.

Finally I ported this to JavaScript so each client could pack the Opus frames in WebM with timestamps correctly starting at 0. This starts the live stream without prebuffering within a second in Chrome and Firefox. Note that I used a 4 byte header on the incoming websocket packets, because the existing distribution system does not preserve packet boundaries (it was built for an MP3 stream). If you use one websocket frame per opus frame and use a fixed number of samples per frame you could remove this header.

Now all that's left is finding a solution for IE11, Safari and some older mobile browsers...