SignalR Owin Self-Host on Linux/Mono SocketException when clients lose connection

Question

I'm running very simple signalr server self-hosted via Owin on ubuntu server 14.04, mono 3.2.8. (code below).

Connecting/Disconnecting works fine on both a remote windows server and when I deploy the bits to the linux server. But when a client dies unexpectedly instead of telling signalr that he's disconnecting, that's when I get a never-ending SocketException only on the linux server only. The windows server disconnects the client after about 30 seconds or so, but the linux server spews the socketexception (also below) every 10 seconds or so, forever.

How can I make the linux server behave like the windows server when running the same code, disconnect the user after a set timeout and not throw socketexceptions?

Server Code:

using System;
using System.Threading.Tasks;
using Microsoft.AspNet.SignalR;
using Owin;

namespace signalrtestserver
{
    class Program
    {
        static void Main(string[] args)
        {
            var uri = System.Configuration.ConfigurationManager.AppSettings["startup_uri"] ?? "http://*:7890";
            using (Microsoft.Owin.Hosting.WebApp.Start<Startup>(uri))
            {
                Console.WriteLine(string.Format("Server started on {0}. Press enter to close.", uri));
                Console.ReadLine();
            }
        }
    }

    class Startup
    {
        static Hub hub;

        public void Configuration(IAppBuilder app)
        {
            app.UseCors(Microsoft.Owin.Cors.CorsOptions.AllowAll);
            var configuration = new HubConfiguration();
            configuration.EnableDetailedErrors = true;
            app.MapSignalR("/signalr", configuration);
            hub = new MyHub();
        }
    }

    public class MyHub : Hub
    {
        public override Task OnConnected() { Console.WriteLine(Context.ConnectionId + " connected"); return base.OnConnected(); }
        public override Task OnDisconnected() { Console.WriteLine(Context.ConnectionId + " disconnected"); return base.OnDisconnected(); }
        public override Task OnReconnected() { Console.WriteLine(Context.ConnectionId + " reconnected"); return base.OnReconnected(); }
    }
}

Client Code:

using System;
using System.Net;
using Microsoft.AspNet.SignalR.Client;

namespace signalrconnection
{
    class Program
    {
        static void Main(string[] args)
        {
            var uri = System.Configuration.ConfigurationManager.AppSettings["signalr_uri"] ?? "http://localhost:7890/signalr";
            ServicePointManager.DefaultConnectionLimit = 10;
            var hubConnection = new HubConnection(uri, false);
            hubConnection.StateChanged += stateChange => Console.WriteLine(string.Format("SignalR {0} >> {1} ({2})", stateChange.OldState, stateChange.NewState, hubConnection.Transport == null ? "<<null>>" : hubConnection.Transport.Name));
            var hubProxy = hubConnection.CreateHubProxy("MyHub");
            hubConnection.Start();
            Console.WriteLine("Press enter to die...");
            Console.ReadLine();
            //hubConnection.Dispose(); //uncomment this to simulate a graceful disconnect which works on both windows and linux
        }
    }
}

Never-Ending Mono Exception:

{path-to-project}/Microsoft.AspNet.SignalR.Core.dll Error : 0 : SignalR exception thrown by Task: System.AggregateException: One or more errors occured ---> System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: Connection reset by peer
  at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
 --> (Inner exception 0) System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: Connection reset by peer
  at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0

Any help would be much appreciated. Thanks in advance!

Answer 1

The problem is this line static Hub hub, and this one hub = new MyHub() in your Startup method.

You do not need to explicitly create instances of the Hub class, neither do you need to keep a reference around. The hub class is instantiated on every request to the server. See http://www.asp.net/signalr/overview/signalr-20/hubs-api/hubs-api-guide-server#transience the section on Hub object lifetime.

Answer 2

This means that whenever there is a disconnection, Mono in Linux is throwing a different exception than MS.NET in Windows. SignalR was implemented by people which were most likely using MS.NET and therefore SignalR must be expecting certain exception and be religious about this.

The best way to fix this is debug the code stepping into SignalR implementation (when running on Linux with Mono) to see where is the exception being caught, and compare to what happens in SignalR under MS.NET in Windows. Then create a minimal testcase about the difference and submit a bug in http://bugzilla.xamarin.com/, they will probably fix it soon (or you can assign it to me and I'll look at it, because I have interest in running SignalR under mono).

Answer 3

I had the same issue on Raspberry Pi with simple Owin self hosted demo app. Requesting "welcome" page from single browser by holding F5 was enough to kill the host within 15 seconds. I have not done any automated stress testing yet, as my "user acceptance" test was failing right away.

The reliable solution I found is to create custom OwinMiddleware to catch and jam the socket exception:

class CustomExceptionMiddleware : OwinMiddleware
{
    public CustomExceptionMiddleware(OwinMiddleware next)
        : base(next)
    {
    }

    public override async Task Invoke(IOwinContext context)
    {
        try
        {
            await Next.Invoke(context);
        }
        catch (IOException ex)
        {
            Console.WriteLine(ex.Message);
        }
    }
}

The Owin startup will need to be updated to use the middleware:

using Owin;

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        app.Use<OwnExceptionMiddleware>()
           .UseNancy();
    }
}

It was inspired by http://dhickey.ie/2014/02/bubbling-exceptions-in-nancy-up-the-owin-pipeline/

Here is my exception for reference:

Unhandled Exception:
System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: The socket has been shut down
  at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.Close () [0x00000] in <filename unknown>:0
  at System.Net.HttpConnection.Close (Boolean force_close) [0x00000] in <filename unknown>:0
  at System.Net.HttpListenerResponse.Close (Boolean force) [0x00000] in <filename unknown>:0
  at System.Net.HttpListenerResponse.Abort () [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse.End () [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End () [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End (System.Exception ex) [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestAsync>d__5.MoveNext () [0x00000] in <filename unknown>:0
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <filename unknown>:0
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestsAsync>d__0.MoveNext () [0x00000] in <filename unknown>:0

Consider being specific in what exceptions you intercept:

    public override async Task Invoke(IOwinContext context)
    {
        try
        {
            await Next.Invoke(context);
        }
        catch (IOException ex)
        {
            if (ex.HResult == -2146233087) // The socket has been shut down
            {
                NLog.LogManager.GetLogger("OwinExceptionHandler").Trace(ex);
            }
            else
            {
                throw;
            }

        }
    }

Answer 4

I think I have the same problem. I described it:

• https://katanaproject.codeplex.com/workitem/438
• https://bugzilla.xamarin.com/show_bug.cgi?id=33254#c4

I've found solution for my situation. I set HttpListener.IgnoreWriteExceptions property to 'true' in OWIN configuration method and then register middlewares. For example:

internal class Startup
{
    public void Configuration(IAppBuilder app)
    {
        object httpListener;

        if (app.Properties.TryGetValue(typeof(HttpListener).FullName, out httpListener)
            && httpListener is HttpListener)
        {
            // HttpListener should not return exceptions that occur
            // when sending the response to the client
            ((HttpListener)httpListener).IgnoreWriteExceptions = true;
        }

        app.Use<TestOwinMiddleware>();
    }
}

I hope this may help you.