SQL Server 2012 Remote connection giving Timeout and ASYNC_NETWORK_IO errors

Question

I am experiencing problems with a service that connects to a remote instance of SQL Server 2012.

The .NET 2.0 service runs on the clients machine and it connects using TCP/IP to a remote web server which is currently running an instance of SQL Server 2012. The web server used to be Windows Server 2003 and SQL Server 2005 and was upgraded 6 months ago without any problems. The service then checks to see if there is any data that it needs to download.

Client Machine

• Windows Server 2003

Web Server

• Windows Server 2012
• SQL Server 2012

This was running fine for 5+ years until last Friday when it suddenly stopped working and it gives the following error log message

Event Type: Warning
Event Source:   
Event Category: None
Event ID:   0
Date:       21/05/2013
Time:       16:45:11
User:       N/A
Computer:   
Description:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadByteArray(Byte[] buff, Int32 offset, Int32 len)
   at System.Data.SqlClient.TdsParserStateObject.ReadUInt32()
   at System.Data.SqlClient.TdsParser.ReadSqlValueInternal(SqlBuffer value, Byte tdsType, Int32 typeId, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ReadColumnData()
   at System.Data.SqlClient.SqlDataReader.ReadColumnHeader(Int32 i)
   at System.Data.SqlClient.SqlDataReader.ReadColumn(Int32 i, Boolean setTimeout)
   at System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32 i)
   at System.Data.SqlClient.SqlDataReader.GetValue(Int32 i)
   at RedBlack.Data.Synchronisation.SyncManager.UpsertRecords(String sourceSelect, String destTable, String[] destFields, List`1 relations)
   at General.CyPro.Application.WebSynchronisation.SyncRoutines.FetchOrders(UpsertFlags upsertOptions)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at RedBlack.CyPro.Application.WebSynchronisation.Job.Run(Object[] args)
...
System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadByteArray(Byte[] buff, Int32 offset, Int32 len)
   at System.Data.SqlClient.TdsParserStateObject.ReadUInt32()
   at System.Data.SqlClient.TdsParser.ReadSqlValueInternal(SqlBuffer value, Byte tdsType, Int32 typeId, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.ReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ReadColumnData()
   at System.Data.SqlClient.SqlDataReader.ReadColumnHeader(Int32 i)
   at System.Data.SqlClient.SqlDataReader.ReadColumn(Int32 i, Boolean setTimeout)
   at System.Data.SqlClient.SqlDataReader.GetValueInternal(Int32 i)
   at System.Data.SqlClient.SqlDataReader.GetValue(Int32 i)
   at RedBlack.Data.Synchronisation.SyncManager.UpsertRecords(String sourceSelect, String destTable, String[] destFields, List`1 relations)
   at General.CyPro.Application.WebSynchronisation.SyncRoutines.FetchOrders(UpsertFlags upsertOptions)

I have checked the activity monitor on the web server and when the query is run it is suspended with a WAIT_TYPE of ASYNC_NETWORK_IO.

I have run the query that it is trying to run manually in Management Studio on the clients server connecting remotely to the web server instance and it runs for about 40 seconds returning 2600+ rows before it gives me the following error

Msg 121, Level 20, State 0, Line 0
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.)

I have run the same query on my machine in Management Studio remotely connecting to the web server instance and it returns the complete 4000 rows in 0.312 seconds.

I am at a loss trying to figure this out as it seems to me to be some sort of network issue. The clients hardware technicians have checked the network and have said that everything is fine as far as they are concerned.

If you need any more information let me know

Thanks

Answer 1

Judging from the resolution by others who have had (somewhat) similar problems, this almost always turns out to be one of three things:

1. Corrupt or failing disk(s). Use CHKDSK /r to address this.

2. A bad NIC, or

3. DHCP problems, particularly with the Lease Expiration/Renewal.

I am not aware of any cases where it turned out to be anything specific to SQL Server.

Answer 2

This has been resolved and the issue in the end turned out to be a problem with the clients internet. I dont have all the details but BT found an issue at the exchange and once this was fixed then the program began working again.