UWP - Streaming WebCam over Socket to MediaElement - Broken Picture?

Question

Background

The code I've written records video clips from a webcam, writes them to a memory stream, then transmits the data across a Socket connection where it's re-assembled into video and played back on a Media Element.

The ultimate goal is to create a baby monitor system, with the server/camera running on a Windows IOT Raspberry Pi, and a UWP app that my girlfriend and I can view on our mobile phones, or the laptop. As well as viewing the camera from another part of the house, we'll also be able to log in when one of us is away from home and in time I'll wire up a PIR motion sensor and alerting system also, but first things first.

The system as a whole work fairly well, there's a 5 second delay in the video which is acceptable to me (for now), and using a MediaPlaybackList the video is streamed at a fairly constant rate with seamless (as seamless as this can get for now) transition between videos. The MediaPlaybackList removes items as they've been played, keeping the memory footprint to a relative constant.

The Issue

When the video plays back on the client end, it gets frequent but random sections of broken picture. There's no pattern to it, not one that I can find anyway, and the only way I can describe it is that part of the picture is split in half horizontally and the two halves are swapped around, the right side of the picture being displayed on the left, and vice versa. It's like a flicker, as in the broken bit is only displayed for a fraction of a second, because another one appears a second or so later somewhere else on the picture.

Here's an example:

Now, here's a couple of interesting points..

1) Before I started using a MediaPlaybackList to queue up streams of data, I was using a method of extracting each video from the incoming socket stream, saving it to the local disk as a StorageFile, then queueing up these StorageFiles, playing them in order and deleting them afterwards (I still have a version of this code in source control which I can dig out, but I don't like the idea of creating and destroying StorageFiles as it seems horrendously inefficient). However, using this method did not result in the broken pictures that I'm now seeing... this leads me to believe that the video itself is fine, and that perhaps it's an issue with the way it's being put back together and streamed to the Media Element?

2) My girlfriend's cat knocked the webcam (a Microsoft Lifecam HD-3000) onto its side without me realising, and I didn't realise until I ran the server and noticed the picture was at a 90 degree angle.. the interesting (and puzzling) thing about this was that the picture delivered to the client didn't break up as I've been describing above. The only difference here that I can see is that the picture then came through as 480 x 640 (from the camera sitting on its side), rather than the standard 640 x 480. What this means, I'm not sure...

Thoughts on the problem

• Something to do with sizing/dimensions of the video (it played fine on it's side, so is it something to do with that)?
• Something to do with bitrate?
• Something to do with the way the bytes are re-assembled at the client?
• Something to do with the encoding of the stream?

Source

Here's a few snippets of code that I think are probably relevant, the full solution source can be found on GitHub, here: Video Socket Server .

Server

while (true)
{
    try
    {
        //record a 5 second video to stream
        Debug.WriteLine($"Recording started");
        var memoryStream = new InMemoryRandomAccessStream();
        await _mediaCap.StartRecordToStreamAsync(MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga), memoryStream);
        await Task.Delay(TimeSpan.FromSeconds(5));
        await _mediaCap.StopRecordAsync();
        Debug.WriteLine($"Recording finished, {memoryStream.Size} bytes");

        //create a CurrentVideo object to hold stream data and give it a unique id
        //which the client app can use to ensure they only request each video once
        memoryStream.Seek(0);
        CurrentVideo.Id = Guid.NewGuid();
        CurrentVideo.Data = new byte[memoryStream.Size];

        //read the stream data into the CurrentVideo  
        await memoryStream.ReadAsync(CurrentVideo.Data.AsBuffer(), (uint)memoryStream.Size, InputStreamOptions.None);
        Debug.WriteLine($"Bytes written to stream");

        //signal to waiting connections that there's a new video
        _signal.Set();
        _signal.Reset();
    }
    catch (Exception ex)
    {
        Debug.WriteLine($"StartRecording -> {ex.Message}");
        break;
    }
}

Connection

//use the guid to either get the current video, or wait for the 
//next new one that's added by the server
Guid guid = Guid.Empty;
Guid.TryParse(command, out guid);
byte[] data = _server.GetCurrentVideoDataAsync(guid);
if (data != null)
    await _socket.OutputStream.WriteAsync(data.AsBuffer());

Client App

byte[] inbuffer = new byte[10000000];

//block on the input stream until we've received the full packet,
//but use the Partial option so that we don't have to fill the entire buffer before we continue.
//this is important, because the idea is to set the buffer big enough to handle any packet we'll receive,
//meaning we'll never fill the entire buffer... and we don't want to block here indefinitely
result = await socket.InputStream.ReadAsync(inbuffer.AsBuffer(), inbuffer.AsBuffer().Capacity, InputStreamOptions.Partial);

//strip off the Guid, leaving just the video data
byte[] guid = result.ToArray().Take(16).ToArray();
byte[] data = result.ToArray().Skip(16).ToArray();
_guid = new Guid(guid);

//wrap the data in a stream, create a MediaSource from it,
//then use that to create a MediaPlackbackItem which gets added 
//to the back of the playlist...
var stream = new MemoryStream(data);
var source = MediaSource.CreateFromStream(stream.AsRandomAccessStream(), "video/mp4");
var item = new MediaPlaybackItem(source);
_playlist.Items.Add(item);

Answer 1

I'm looking to do something similar (stream video/audio from a UWP app on a Raspberry Pi) but I have been using the simple-communications sample from the Windows 10 SDK which after a bit of tweaking I have been able to get working reliably (there are thread sync issues with the sample code). However the SDK sample uses a proprietary protocol using media extensions and it isn't easy to redirect the stream over the internet which is my use-case so I had a look at your code and got it working (with the same bugs). Simple Real Time communication

A couple of comments on your approach:

1) The RPi can't process video on Win10 very well as it can't use the hardware video encoders so does everything in software. This will cause glitches and I see the CPU performance increasing significantly with over 50% utilisation which means at least one of the CPU cores is working close to max, possibly the one handling the video compression to MP4. However I ran up the SDK sample and got glitch free viewing and about 70% CPU so your problem is likely elsewhere.

2) 5 seconds of latency delay is significant. I get less than 100mSec latency with the real time sample however when I adjusted down your streaming timer to 1 second the breakup was significant and unworkable. Have you thought about changing the design so it streams during capture however I'm not sure the InMemoryRandomAccessStream will let you do that. Another alternative is to capture the preview stream and write a custom media sink to buffer (harder to do as not managed code and likely not able to compress as easily) like the Simple Communication sample does.

3) MP4 is a container not a compression format, and isn't built for streaming as the whole file has to be downloaded before it starts unless the moov metadata record is placed at the beginning of the file. Not sure how UWP handles this, likely your approach of closing off the stream before sending is required to ensure the other end can play it properly.

So not a complete answer but hopefully the above helps.