Streaming WebCamTexture with new Unity 5.1 Transport Layer API

Question

As the question states, I have been trying to stream a WebCamTexture from a client with a webcam to a server. Both sides (client and server) are in Unity. Later, the client will be deployed to Android and the server will be a desktop application.

Currently I am getting the pixels of the texture using:

tex.GetPixels32();

and Serializing them with a custom serializer (to optimize its size). I currently have an uncompressed byte array with around 3,5MB per frame ready to be sent. I know it is huge, but I wanted to have it transmitted before start with the compression part and real time part.

The last part of the process should be to send it using the new Unity NetworkTransport static class. It has been a long time since the last time I used sockets and I am really rotten. Currently I am unable to have it working.

Here it is my code for the server side (omitting serialization code for clarity):

void Start()
{
    webcamTexture = new WebCamTexture();
    Background.texture = webcamTexture;
    Background.material.mainTexture = webcamTexture;
    webcamTexture.Play();

    if (!_isStarted)
    {
        _isStarted = true;

        NetworkTransport.Init();
        m_Config = new ConnectionConfig();
        m_CommunicationChannel  = m_Config.AddChannel(QosType.ReliableFragmented);
        HostTopology topology = new HostTopology(m_Config, 12);
        m_GenericHostId = NetworkTransport.AddHost(topology, 0);
        byte error;
        m_ConnectionId = NetworkTransport.Connect(m_GenericHostId, ip, port, 0, out error);
    }
}

void Update()
{
    if (!_isStarted)
        return;

    NetworkEventType recData = NetworkTransport.Receive(out recHostId, out connectionId, out channelId, recBuffer, bufferSize, out dataSize, out error);

    switch (recData)
    {
        case NetworkEventType.Nothing:         //1
            break;
        case NetworkEventType.ConnectEvent:    //2
            Debug.Log("Received connection confirmation");
            _readyToSend = true;
            break;
        case NetworkEventType.DataEvent:       //3

            break;
        case NetworkEventType.DisconnectEvent: //4
            //one of the established connection has been disconnected
            Debug.Log(String.Format("Disconnect from host {0} connection {1}", recHostId, connectionId));

            break;
    }

    if (_readyToSend)
    {
        _readyToSend = false; // To send just the first frame

        byte[] colourArray = SerializeObject(MakeSerializable(GetRenderTexturePixels(webcamTexture))); // Serialize the webcam texture

        // Sending total size
        byte[] sizeToSend = BitConverter.GetBytes(colourArray.Length);
        NetworkTransport.Send(m_GenericHostId, m_ConnectionId, m_CommunicationChannel, sizeToSend, sizeToSend.Length, out error);

        byte[] bytes = new byte[bufferLenght];
        int remainingBytes = colourArray.Length;
        int index = 0;
        int i = 1;

        while (remainingBytes >= bufferLenght)
        {
            System.Buffer.BlockCopy(colourArray, index, bytes, 0, bufferLenght);
            NetworkTransport.Send(m_GenericHostId, m_ConnectionId, m_CommunicationChannel, bytes, bytes.Length, out error);
            remainingBytes -= bufferLenght;
            Debug.Log(i++ + "Remaining bytes: " + remainingBytes + " - Error: "+error);
            index += bufferLenght;
        }

        if (remainingBytes > 0) // Send the last fragment below bufferLenght bytes
        {
            System.Buffer.BlockCopy(colourArray, index, bytes, 0, remainingBytes);
            NetworkTransport.Send(m_GenericHostId, m_ConnectionId, m_CommunicationChannel, bytes, remainingBytes, out error);
            Debug.Log("Error: "+error);
        }
    }
}

And this is the client side:

void Start()
{
    if (!_isStarted)
    {
        _isStarted = true;

        NetworkTransport.Init();

        m_Config = new ConnectionConfig();

        m_CommunicationChannel  = m_Config.AddChannel(QosType.ReliableFragmented);

        HostTopology topology = new HostTopology(m_Config, 12);

        m_GenericHostId = NetworkTransport.AddHost(topology, port, null);
    }
}

void Update()
{
    if (!_isStarted)
        return;

    int recHostId; 
    int connectionId; 
    int channelId; 
    byte[] recBuffer = new byte[bufferLenght]; 
    int bufferSize = bufferLenght;
    int dataSize;
    byte error;

    NetworkEventType recData = NetworkTransport.Receive(out recHostId, out connectionId, out channelId, recBuffer, bufferSize, out dataSize, out error);

    switch (recData)
    {
        case NetworkEventType.Nothing:         //1
            break;

        case NetworkEventType.ConnectEvent:    //2

                //somebody else connect to me
            Log.text += string.Format("Connect from host {0} connection {1}\n", recHostId, connectionId);
            break;

        case NetworkEventType.DataEvent:       //3

            if (!sizeReceived)
            {
                sizeReceived = true;

                if (dataSize == 2)
                {
                    bytesToReceive = BitConverter.ToInt16(recBuffer, 0);
                }
                else if (dataSize == 4)
                {
                    bytesToReceive = BitConverter.ToInt32(recBuffer, 0);
                }

                Debug.Log("We will receive: "+bytesToReceive);
            }
            else
            {
                Log.text = string.Format("Received event host {0} connection {1} channel {2} message length {3}\n", recHostId, connectionId, channelId, dataSize);

                Log.text += "Received " + bufferSize + " bytes\n";
                bytesToReceive -= bufferSize;
                Log.text += "Remaining " + bytesToReceive + " bytes\n";
            }
            break;

        case NetworkEventType.DisconnectEvent: //4

            break;

    }
}

I know it will block the Update function until sent, but it is not important right now to me, since I am just trying to get a frame transmitted to understand how this new system works, and proceed from there. Currently I am getting this error after the first package sent, with a buffer of 32768 bytes:

no free events for long message
UnityEngine.Networking.NetworkTransport:Send(Int32, Int32, Int32, Byte[], Int32, Byte&)
CameraStreamer:Update() (at Assets/Scripts/Client/CameraStreamer.cs:112)

I have also tried to use a 1024 buffer, and it only takes longer to show the message (after more than 100 successfully sent packages).

According to this thread, it is related to messages being sent too fast and filling the queue, but none of the proposed solutions worked for me. I would appreciate any help or orientation, since Unity documentation is really poor.

Answer 1

This particular test code / example will still have issues even with the Unity 5.2 patch.

Running the code in Unity 5.3.4f1, I was able to see that error 4 (NetworkError.NoResource) occurs after some 200 packets and stops sending shortly afterwards. I presume the reason is because it was a blocking send and so the message queue never flushes properly.

I've rewritten the code to grab the webcam image off the texture after 2 second delay (because the webcam needs time to intialise, otherwise you send a blank image).

Afterwards sending through the webcam image, a single packet per execution of Update now seems fine. Probably because Update had time to run and was not blocked. So send a packet and exit the Update function, moving the index and other variables out of Update.

Hope this helps anyone else looking at this, took me 3 days to figure out.

EDIT: For testing, a more simple method is to move out the blocking send (while bit) and add it into a co-routine. Seems to work that way as well.

Answer 2

I finally managed to have the camera screenshot sent over the network. It seems that try to send several messages without waiting for the server to answer kind of fill the queue.

All I needed to do was to send an answer back from the server to the client with a single byte after every message received. The client is waiting for this kind of ACK to send the next message. With this, everything started working like a charm.

I think that there must be an alternative way to solve this issue, since I don't see very reasonable to send an message back for every message received, but it did the trick for now. I will edit this answer with additional findings.

Regards.

EDIT: It turned out to be an Unity bug, and it was solved in Unity 5.2. (if I recall correctly).