create and transmit custom mediastream in webrtc

Question

I want to use canvas element as mediastreamsource of video part of the webrtc communication, any directions would be helpful, scoured the net, not finding much resources discussing this topic

* Long Background Story *

The problem, I cannot send the video from camera directly, it is part of the requirements that I process the video(some image processing stuff, out of scope for this issue) before displaying.

Previously, on the other peer's browser, instead of directly displaying the video using a <video> tag, I did some processing on a hidden canvas element, and then copied the details to another canvas (I used a settimeout to keep drawing, which gave the illusion of live video).

Now, the client wants the processing done before transmission of video, so I used webrtc to pass the audio stream directly (previously both audio and video were sent via webrtc). For the video stream, I had two solutions:

Steps:

1. Process video on local peer, draw on hidden canvas. the easy part.

2. Use timeout to repeatedly capture image data and transmit
   a) using websockets( yes, goes through server), which came with a horrible lag and an eventual crash of the browser.
   b) using RTCDataChannel, which had much better performance, but at times fails for no reason. I also had several other issues (e.g. the extra bandwidth used, because of sending jpeg instead of webp).

Another major is issue is that because I am using timeout: when I switch tabs, the frame rate drops on the other side.

So, is there any way I can use the hidden canvas as mediastreamsource instead of me doing it manually?

Answer 1

mozCaptureStreamUntilEnded is going to be the basis for a proposal Martin Thompson is working on for the WG, to connect directly to a MediaStream. A workaround in Firefox per the comments here is mozCaptureStreamUntilEnded from a fed from a canvas captured from the MediaStream. An ugly sequence, which is part of why we're going to allow direct output of a to a MediaStream (and standardize captureStream on as well).

Note that feeding mozCaptureStream(UntilEnded) to a PeerConnection was broken for a while (partly since it's non-standard thus far); it's fixed in Firefox 36 (due on the release channel in 6 weeks; going to Beta next week). See Bug 1097224 and Bug 1081409

And incredibly hacky way on Chrome and Firefox would put the video in a window, then screencapture the window. I don't advise it since it requires screensharing permission, selecting the window, etc.

The only other option for Chrome (or Firefox) is to save frames of video as JPEGs (as you mention) and send over a DataChannel. Effectively Motion-JPEG, but run by JS. Framerate and quality (and delay) will suffer. You may want to use an unreliable channel since on errors you can throw the frame away and just decode the next one (it's MJPEG after all). Also, if delay gets too high, reduce the frame size! You'll want to estimate the end-to-end delay; best way is to feed back the decode time over datachannels to the sender and have it use the reception time of that packet to estimate delay. You care more about changes in delay than absolute values!!