SqlException Transaction was deadlocked on communication buffer resources

Question

I have a server process that has to execute a lot of database queries, it uses TPL to run stuff in parallel. It has been working fine for all of this year, until today when it crashed twice in a 30 minute span with the following exception:

Transaction (Process ID 89) was deadlocked on communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

The database is configured to log any deadlocks, but it didn't log anything, so it seems as if this deadlock happened only on the client side?

I couldn't find any references to this exception other than one msdn forum post that doesn't provide any information.

Has anyone seen this exception before? Or know what I could do to find out more information about it?

---> System.AggregateException: One or more errors occurred. 

---> System.Data.SqlClient.SqlException: Transaction (Process ID 89) was deadlocked on communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at App.CoreEngine.V5.DataAccess.SqlReader.Read(String readerDescription) in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\SqlReader.cs:line 121
   at App.CoreEngine.V5.DataAccess.DataContext.ExecuteQuery(PtQuery query, ValueStore`1 store, String readerDescription) in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 328
   at App.CoreEngine.V5.DataAccess.DataContext.<>c__DisplayClass12.<GetCalculatedDataForCompare>b__f(Object _) in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 267
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at App.CoreEngine.V5.DataAccess.DataContext.GetCalculatedDataForCompare() in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 276
   at System.Threading.Tasks.Task`1.InvokeFuture(Object futureAsObj)
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task`1.get_Result()
   at App.CoreEngine.V5.DataAccess.DataContext.get_CalcCompareData() in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 389
   at App.CoreEngine.V5.Calculation.CalculationEngine.Run() in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\Calculation\CalculationEngine.cs:line 243
   at App.CoreEngine.V5.Processor.Milestone.BatchRunner.Run() in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\Processor\Milestone\BatchRunner.cs:line 171

---> (Inner Exception #0) System.AggregateException: One or more errors occurred. 
---> System.Data.SqlClient.SqlException: Transaction (Process ID 89) was deadlocked on communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.HasMoreRows()
   at System.Data.SqlClient.SqlDataReader.ReadInternal(Boolean setTimeout)
   at App.CoreEngine.V5.DataAccess.SqlReader.Read(String readerDescription) in C:\SourceCode\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\SqlReader.cs:line 121
   at App.CoreEngine.V5.DataAccess.DataContext.ExecuteQuery(PtQuery query, ValueStore`1 store, String readerDescription) in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 328
   at App.CoreEngine.V5.DataAccess.DataContext.<>c__DisplayClass12.<GetCalculatedDataForCompare>b__f(Object _) in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 267
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at CoreEngine.V5.DataAccess.DataContext.GetCalculatedDataForCompare() in C:\SourceCode\AppV1\Releases\Libraries\CoreEngine\CoreEngine.V5\DataAccess\DataContext.cs:line 276
   at System.Threading.Tasks.Task`1.InvokeFuture(Object futureAsObj)
   at System.Threading.Tasks.Task.Execute()

EDIT - I just double checked that dbcc 1222 is turned on for the server this happened on, I ran dbcc tracestatus and got:

TraceFlag   Status  Global  Session
1222    1   1   0
3605    1   1   0

And there is nothing in the logs reporting the deadlocks

Answer 1

My guess is that the execution plan now uses parallelism, whereas before it didn't meet the cost threshold.

Try MAXDOP 1 in the queries

Edit, after comment

You also need trace flag 1204.

TF 1222 gives the deadlock graph but with this "communication buffer resource" deadlock there may not be 2 objects involved (I'm guessing here it isn't an index/table conflict). See http://msdn.microsoft.com/en-us/library/ms178104.aspx

There is also the undocumented TF 1205 which gives more info into the error log

Answer 2

Are you able to try any of the following:

• Leave a server side trace running to capture deadlock events
• Apply CU1 and try Retrieving Deadlock Graphs with SQL Server 2008 Extended Events
• Switch on the other deadlock capture trace flag, 1205