Microsoft Azure Service Bus Timeout Exceptions

Question

We have an application that runs on the Microsoft Azure cloud platform. The communication between some components is made using Service Bus. Everything was working fine, until recently we started to get the following type of timeout exceptions:

When calling QueueClient x.Send(...)

Exception rethrown at [0]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory `1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

When calling NamespaceManager x.GetQueue(...)

PROGRESS queue processing failed. System.TimeoutException: The request has timed out after 60000 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. TrackingId:bdffb6bd-5367-4573-aaa3-8ea9a03f5a2b,TimeStamp:5/28/2015 8:39:46 AM ---> System.Net.WebException: The request was aborted: The request was canceled. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.ServiceBus.Messaging.ServiceBusResourceOperations.GetAsyncResult`1.b__49(GetAsyncResult`1 thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

When calling NamespaceManager x.SubscriptionExists(...)

Exception doing periodic work: System.TimeoutException: The request has timed out after 00:10:00 milliseconds. The successful completion of the request cannot be determined. Additional queries should be made to determine whether or not the operation has succeeded. Server stack trace: Exception rethrown at [0]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.NamespaceManager.OnEndSubscriptionExists(IAsyncResult result) at Microsoft.ServiceBus.NamespaceManager.SubscriptionExists(String topicPath, String name) ...

When calling QueueClient x.Receive(...)

PROGRESS queue processing failed. Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Error during communication with Service Bus. Check the connection information, then retry. ---> System.ServiceModel.CommunicationObjectFaultedException: Internal Server Error: The server did not provide a meaningful reply; this might be caused by a premature session shutdown. TrackingId:04ba0220-0350-4806-9c65-c2bba9671054, Timestamp:28.05.2015 13:00:55 Server stack trace: Exception rethrown at [0]: at Microsoft.ServiceBus.Common.ExceptionDispatcher.Throw(Exception exception) at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.DuplexCorrelationAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory`1.RequestSessionChannel.RequestAsyncResult.b__4(RequestAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result) ….

The exceptions are clearly related to ServiceBus and they are non-deterministic. The functions throwing them, e.g., Send, GetQueue, SubscriptionExists, are called no more than 100-120 times per minute. We changed nothing in the code and increasing the timeout values (even to ridiculously high values, like 10min) did not help. Also, we do not believe it is some network related problem (on our side) since the same error occur when the application is run from different places.

Has anyone else encountered recently these kind of exceptions? Is there a problem on the side of Microsoft or we are missing something?

Answer 1

A few weeks back we had sudden and unexplainable timing issues crop up with our Service Bus app that has been in production for many months. Ours continued to work but every few calls would take 10+ seconds when they are normally 100-200 millisconds. This went on for a couple weeks and I spent most of that trying to figure out what was going on and never did as the problem suddenly vanished.

We did learn that new Service Bus namespaces we created in the same and other data centers for testing while the problem was occurring did not exhibit the same issue. The Service Bus group offered no help and would only say response times are not guaranteed only the SLA is.