Apache reverse proxy strange timeouts

Question

Problem
I have an apache server in the DMZ setup as a reverse proxy to our live machine. Most of the time this is working like a charm. The problem is the users say that very occasionally they get "a timeout".

Sure enough if I check the error logs on the proxy I can see a handful of these errors.

[error] [client x.x.x.x] (70007)The timeout specified has expired: ZSRV_MSG158A: Beim Lesen eines Dokuments auf dem fernen Host x.x.x.x ist ein Fehler aufgetreten. 

Or in English, When reading a document on the remote host x.x.x.x An error has occurred.

The error logs on the live machine are totally empty.

Things I've tried
My first thought was that maybe the live machine was returning but the proxy timeout was lower so perhaps it had already given up waiting. I found that the live machine has a timeout of 300 seconds (the default) so I tried changing the proxy timeout to 310 seconds.

This hasn't made a difference. This leads me to believe that perhaps there is an intermittent connection problem between the proxy and the server? It's as if the live machine never receives the request.

I've already turned keepalive off on both the live machine and the proxy. I also changed it to use http 1 as apparently this gives less problems?

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#envsettings

Any ideas?

Answer 1

We found the source of the timeouts. There were record locks in a third party API which caused some requests to hang. Once this was fixed we stopped getting timeouts on the proxy.

What confused me was that we had timeout errors on the proxy but not on the main live machine (the proxy destination). This made me think that the request couldn't have timed out on live but somehow only on the proxy. Apparently this was wrong as the source of the timeouts was on the live machine, we just didn't see the apache errors there.

I still don't know why we only saw the timeout errors in the proxy logs so if anyone has an explanation I will accept that as the answer.