Implementing a media device the browser can consume

Question

I'd like to create a media device intended for browser consumption. That is to say, I want to publish a video stream the browser can get through navigator.mediaDevices, send through WebRTC, put in a <video> tag.

In reality, what I'm doing is consuming a video stream (I glean this from a loop in C++ which spits out images), reading it and analyzing it, and I want to be able to send the stream to the browser. Ideally, I would like to be able to do this from a Docker container. The C++ process is also going to coexist as a Node binding, but I'm not sure if that's relevant. In this case, what I'm saying is that, if it's easiest to send the images/video stream through the binding's API and then publish from Node, I have no problem with that.

Can anyone provide documentation or reading material on how to register a faux device with wherever the browser is getting devices from? I'm not very familiar with drivers or anything.

While I think any good solution is going to be fairly cross-compatible with other systems, I only strictly need it to be compatible with Ubuntu 16.04 and Chrome.

Answer 1

The architecture I ended up with is as follows:

C++ using FFmpeg (libavdevice/libavcodec/libavformat etc.) feeds into the device, which I created using v4l2loopback. Then Chrome can detect this pseudo-device. (as long as you use exclusive_caps=1 option as shown below)

So the first thing I do is set up the v4l2loopback device. This is a faux device that will output like a normal camera, but it will also take input like a capture device or similar.

git clone https://github.com/umlaeute/v4l2loopback
cd v4l2loopback
git reset --hard b2b33ee31d521da5069cd0683b3c0982251517b6 # ensure v4l2loopback functionality changes don't affect this script
make
sudo insmod v4l2loopback.ko exclusive_caps=1 video_nr=$video_nr card_label="My_Fake_Camera"

The browser will see the device in navigator.mediaDevices.enumerateDevices() when and only when you're publishing to it. To test that it's working before you feed to it through C++, you can use ffmpeg -re -i test.avi -f v4l2 /dev/video$video_nr. For my needs, I'm using Puppeteer, so it was relatively easy to test, but keep in mind a long-lasting browser session will cache the devices and refresh them somewhat infrequently, so make sure test.avi (or any video file) is quite long (1 min+) so you can try to reset your environment fully. I've never figured out what the caching strategy is exactly, so Puppeteer turned out to be very helpful here, but I had already been using it, so I didn't have to set it up. YMMV.

Now (for me) the hard part was getting FFmpeg (libav-* version 2.8) to output to this device. I can't/won't share all my code, but here are the parts and some guiding wisdom:

Set up:

• Create an AVFormatContext using avformat_alloc_output_context2(&formatContext->pb, NULL, "v4l2", "/dev/video5")
• Set up the AVCodec using avcodec_find_encoder and create an AVStream using avformat_new_stream
• There are a bunch of little flags you should be setting, but I won't walk through all of them in this answer. This snippet as well as some others include a lot of this work, but they're all geared towards writing to disk rather than to a device. The biggest thing you need to change is creating the AVFormatContext using the device rather than the file (see first step).

For each frame:

• Convert your image to the proper colorspace (mine is BGR, which is OpenCV's default) using OpenCV's cvtColor
• Convert the OpenCV matrix to a libav AVFrame (using sws_scale)
• Encode the AVFrame into an AVPacket using avcodec_encode_video2
• Write the packet to the AVFormatContext using av_write_frame

As long as you do all this right, it should feed it into the device and you should be able to consume your video feed in the browser (or anywhere that detects cameras).

---

The one thing I'll add that's necessary specifically for Docker is that you have to make sure to share the v4l2 device between the host and the container, assuming you're consuming the device outside of the container (which I am). This means you'll run the docker run command with --device=/dev/video$video_nr.