Close all underlying network connections

Question

Background:
I have some code that is connecting to a server to pull data. However, at some point the program got into a state where it was failing to connect to the server. The problem we were seeing is that the server was continually timing out when we were attempting to reconnect.

Plugin:
We have our timeout set to the default values used by our network plugin (a Thrift plugin). These values are 100k millisecond timeout and 300k millisecond readwrite timeout. Not exactly short timeouts. Also, every time we attempt to reconnect, we recreate the plugin's class, so whatever values it's setting internally are reset each time we attempt to reconnect. I believe timeouts are not the issue. And I believe the plugin is not the issue.

Network:
We saw these repeated timeouts occur for half an hour. When we restarted the service (not the machine), it immediately came back online. So, I know it wasn't a network issue. That leads me to believe it's something in our code that is getting into an invalid network state. Also, it is a state that we can't control, since our plugin hides all of the good network stuff from us--including timeouts, keepalive flags, and connection groups (among others). We basically don't have access to the HTTPWebReqest member.

Question:
When our network settings get into this state, we attempt to close all connections and reconnect to the server. The goal was to kill all the active connections in order to reset whatever state we got into. However, when we attempt to reconnect, we're getting timeouts on our reconnect.

Somehow (probably due to the KeepAlive and TCP Pipelining, which we can't control), the network connections remain open even though we have closed all connections. This leaves us in a bad state and prevents us from cycling our connections to the server.

The question:
How can I kill all underlying connections to a server in order to force a reconnect?

Answer 1

The underlying .Net architecture will maintain the connection (via the KeepAlive) in order to improve overall network performance. The easiest solution is to just set the KeepAlive to false on all your connections so that they will not maintain those invalid states. The downside is that you'll increase your traffic since they will have to re-establish connections every time.

Also, since you do not have access to the keepalive (or the HTTPWebRequest object), you'll have to find a way to kill the connections outside of the request settings.

There are two approaches you can take to get .Net to release these connections. Both of them involve getting down to the ServicePoint level and dealing with connection groups.

1. CloseConnectionGroups

   When you create your HTTPWebRequest, you can give it a connection group. From there, you can use the CloseConnectionGroup() function from the ServicePoint in order to kill all connections to that service.

   ServicePoint srvrPoint = ServicePointManager.FindServicePoint(serverUri);
   srvrPoint.CloseConnectionGroup(myConnGroup)
   

   Of course, if you have more than one connection group, you'll need to do that in a loop for every connection group.

   Note that if you choose this path, be careful not to put all connections in the same group. This could cause you to run out of sockets. More information here.

2. ReleaseAllConnectionGroups

   There is an internal function within the ServicePoint class that you can call that will kill all connections to the give service point (ie server). However, because it's internal to the class, you'll have to use reflection to get at it:

       ServicePoint srvrPoint = ServicePointManager.FindServicePoint(serverUri);
       MethodInfo ReleaseConns = srvrPoint.GetType().GetMethod
               ("ReleaseAllConnectionGroups",
               BindingFlags.Instance | BindingFlags.NonPublic);
       ReleaseConns.Invoke(srvrPoint, null);
   

   This code just pulls out the ReleaseAllConnectionGroups from that ServerPoint and invokes it with no parameters. This function will go through all connection groups in its internal list and release all of those resources--killing all of the connections.

   Downfall: Because this is a private member, you're not guaranteed that it will be there if you upgrade your version of .Net. It's currently available in versions 2.0 through 4.5, though. A link to the reverse engineered code of ReleaseAllConnectionGroups.

Because you cannot set the connection group, you will have to use option 2.

Unfortunately, there's no other way to get .Net to close those low-level connections and release their KeepAlive status without setting the KeepAlive flag.