Why am I getting this result (same thread entering twice before exiting)?

Question

I've inherited a socket callback routine I'm trying to debug. It does asynchronous TCP communication with an EndReceive() immediately followed by a BeginReceve() so it's always listening. It's filled with "when hit" breakpoints that display a message and continue for diagnostics.

Because it's an asynchronous callback it may be called by the system multiple times for one BeginReceive() even if it's in the middle of processing a previous callback. Usually this is in a new thread, which is why I have a lock (a critical section) for the whole thing. But sometimes it seems to get reentered by the same thread even before it's exited. How is this possible? What am I doing wrong?

He's my trace (using 'When Hit' breakpoints) . . .

Socket_DataArrival() – Entering … managedThreadID=11
Socket_DataArrival() - first statement in lock . . . managedThreadID=11
Socket_DataArrival() - EndReceive() ... managedThreadID=11
Socket_DataArrival() - BeginReceive() . . . managedThreadID=11
Socket_DataArrival() – Entering … managedThreadID=11
Socket_DataArrival() - first statement in lock . . . managedThreadID=11
Socket_DataArrival() - EndReceive() ... managedThreadID=11
Socket_DataArrival() - BeginReceive() . . . managedThreadID=11
Socket_DataArrival() - exiting at end of routine . . . managedThreadID=11
Socket_DataArrival() - exiting at end of routine . . . managedThreadID=11

and here's the routine (proprietary code commented out) . . .

   private void Socket_DataArrival(IAsyncResult ar)
   {
       StateObject stateObject;
       int bytesReceived;
       int managedThreadId = Thread.CurrentThread.ManagedThreadId;

       lock (inputLock)
       {                 // "Entering..."
           try
           {
               _Receiving = false;  //"first statement in lock"
               stateObject = (StateObject)ar.AsyncState;
               bytesReceived = stateObject.sSocket.EndReceive(ar);    
               _Receiving = true;

               _StateObject = new StateObject(2048, _TCPConn);  //2048 = arbitrary number
               _StateObject.tag = "Socket_DataArrival ";

               _TCPConn.BeginReceive(
                    _StateObject.sBuffer,
                    0,
                    _StateObject.sBuffer.Length,
                    SocketFlags.None,
                    new AsyncCallback(Socket_DataArrival),
                    _StateObject);

            }
            catch (Exception exc)
            {
                subs.LogException("Socket_DataArrival", exc);
            }

            // proprietary stuff goes here

        }  // end critical section
        return;
    }

inputLock is defined at the class level as . . .

private Object inputLock = new Object();

How can the same thread enter it a second time before it's exited the first time?

Answer 1

It appears that when you call BeginReceive() and data is already available you get a callback immediately and on the thread that called BeginReceive().

This sounds completely plausible given that the underlying I/O model is almost guaranteed to be I/O completion port based and on operating systems after Windows XP (so all currently supported OSs) you can tell IOCPs to 'skip completion port processing on success' and instead return async data immediately to the caller.

So, what I assume is happening is that the BeginReceive() calls down into WSARecv() and this completes immediately because data is available and so the calling code executes the callback immediately on the calling thread. If there wasn't data available then the WSARecv() would return IO_PENDING and the I/O would eventually complete (when data arrived) and one of the threads that is associated with the socket's IOCP would then deal with the completion and call the handler.

Some data flow patterns are more likely to cause this to occur than others, and, of course, it's dependent on the network and how the data is flowing. It's also much more likely to happen with async sends...

The best way to fix it is to have your I/O handlers simply place all completions into a queue which only one thread can process and which can't be processed recursively. This then leaves you with a design whereby this problem can't happen. I wrote about such a design here, for the ACCU publication Overload.

Answer 2

A lock only prevents a different thread from entering the critical section, not the same one.

Since you call _TCPConn.BeginReceive within Socket_DataArrival with same same callback (also Socket_DataArrival), this callback is simply entered again for when receiving data from _TCPConn.

Try to use a different callback for _TCPConn or try to call BeginReceive() outside of Socket_DataArrival().