What causes a NullReferenceException in .NET Threading / accepting TCP connections?

Question

In my own webserver software, I am getting entries in the Event Viewer on the server that contain the following stack trace:

Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentNullException
Stack:
   at System.Net.FixedSizeReader.ReadCallback(System.IAsyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Net.ContextAwareResult.Complete(IntPtr)
   at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)

Needless to say, this crashes the process, which means the server goes down.

Since none of the stacktrace mentions my own code, I am quite puzzled. Is this a bug in .NET? If so, are there any known workarounds? Or is there a known cause to this particular exception?

The name CompletionPortCallback mentioned in the stacktrace leads me to believe that it occurs when the server tries to accept an incoming TCP connection, so I’m going to include the relevant code for that below. Of course I’m happy to include other code if you think the problem lies elsewhere.

The call to BeginAccept looks like this:

_listeningSocket.BeginAccept(acceptSocket, null);

Here, _listeningSocket is of type System.Net.Sockets.Socket.

The acceptSocket method is shown below. I am going to assume that the comments explain the code well enough; if not, I’m happy to clarify in a comment. Since this code runs in RELEASE mode on the live server, the #if DEBUG will of course be false.

private void acceptSocket(IAsyncResult result)
{
#if DEBUG
    // Workaround for bug in .NET 4.0 and 4.5:
    // https://connect.microsoft.com/VisualStudio/feedback/details/535917
    new Thread(() =>
#endif
    {
        // Ensure that this callback is really due to a new connection (might be due to listening socket closure)
        if (!IsListening)
            return;

        // Get the socket
        Socket socket = null;
        try { socket = _listeningSocket.EndAccept(result); }
        catch (SocketException) { } // can happen if the remote party has closed the socket while it was waiting for us to accept
        catch (ObjectDisposedException) { }
        catch (NullReferenceException) { if (_listeningSocket != null) throw; } // can happen if StopListening is called at precisely the "wrong" time

        // Schedule the next socket accept
        if (_listeningSocket != null)
            try { _listeningSocket.BeginAccept(acceptSocket, null); }
            catch (NullReferenceException) { if (_listeningSocket != null) throw; } // can happen if StopListening is called at precisely the "wrong" time

        // Handle this connection
        if (socket != null)
            HandleConnection(socket);
    }
#if DEBUG
    ).Start();
#endif
}

Answer 1

The answer is as simple as it is disappointing.

It turns out that the ArgumentNullException was indeed thrown by my own code, which executed in the callback to the async call and should have been in the stack trace. I trusted the stack trace too much; the fact that this didn’t show in the stack trace led me on a wrong track for a long time.

As C# developers know, the throw; statement (not throw e;) is supposed to keep the exception stack trace unaltered. System.Net.FixedSizeReader.ReadCallback catches and rethrows the exception via such a throw; statement, yet the stack trace shown in the Event Viewer became truncated. I can only surmise that this is a bug in the CLR or the Event Viewer or some interaction between the two, causing only the part of the stack trace from the throw; instruction onwards to show.

When I ran the software on the console instead of as a service, which I should have thought of much sooner, the full stack trace of the exception showed on the console, indicating that the true cause of the exception was my own code.