WCF Service calls always fail after 30 seconds with (502) Bad Gateway

Question

We have a WCF service (BasicHttpBinding) which will always fail after 30 seconds. Calls under 30 seconds complete without error. Anything over 30 seconds will fail with a 502 Bad Gateway exception:

System.Net.WebException: The remote server returned an error: (502) Bad Gateway.

But yet the WCF call continues to run in the background (and will eventually complete). We have confirmed that the BasicHttpBinding - Binding - sendTimeout (in web.config) is greater than 30 seconds (actually set to 5 minutes). We confirmed this both on the client and the server.

Here is the full stack trace:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (502) Bad Gateway. ---> System.Net.WebException: The remote server returned an error: (502) Bad Gateway.
   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace: 
   at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory factory, WebException responseException, ChannelBinding channelBinding)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

Any ideas where this 30 second "timeout" is coming from or why a 502 Bad Gateway error is returned?

SOLUTION: We are using the IIS7 Application Request Routing Module which has Proxy settings of its own. The Proxy settings have a default timeout of 30 seconds. Increasing this to 600 seconds (10 minutes) solved our problem. The Bad Gateway error is not completely correct but WCF Trace Viewer (see answer) helped see that the problem was not the service itself but an issue in between the client and the wcf service.

Answer 1

You may want to look try modifying the other timeout config values:

closeTimeout, openTimeout, recieveTimeout.

See this MSDN post for information on the config items, summary below:

Client side:

• SendTimeout is used to initialize the OperationTimeout, which governs the whole interaction for sending a message (including receiving a reply message in a request-reply case). This timeout also applies when sending reply messages from a CallbackContract method.
• OpenTimeout and CloseTimeout are used when opening and closing channels (when no explicit timeout value is passed).

Server side:

• Send, Open, and Close Timeout same as on client (for Callbacks).
• ReceiveTimeout is used by ServiceFramework layer to initialize the session-idle timeout.

ADDED:

The only other thing I can suggest is to use the WCF Service Trace Viewer to get to the bottom of what's causing the issue. See this SO Post if you require details on how to use it.