haproxy 504 timeout to apache

Question

Very new to haproxy and loving it, apart from a 504 issue that we're getting. The relevant log output is:

Jun 21 13:52:06 localhost haproxy[1431]: 192.168.0.2:51435 [21/Jun/2017:13:50:26.740] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 13:54:26 localhost haproxy[1431]: 192.168.0.2:51447 [21/Jun/2017:13:52:46.577] www-https~ beFootprints/foorprints 0/0/3/-1/100005 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:15:57 localhost haproxy[1431]: 192.168.0.1:50225 [21/Jun/2017:14:14:17.771] www-https~ beFootprints/foorprints 0/0/2/-1/100004 504 195 - - sH-- 3/3/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:22:26 localhost haproxy[1431]: 192.168.0.1:50258 [21/Jun/2017:14:20:46.608] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1" 

Using the following timeout values in the haproxy.cfg

defaults
        log     global
        mode    http
        option forwardfor
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  100000

Running on Ubuntu 16.04.2 LTS

Any help and comment very much appreciated!

Answer 1

The problem appears to be with the web server. Check the logs, there, and you should find long-running requests.

Here's how I conclude that.

Note sH-- in your logs. This is the session state at disconnection. It's extremely valuable for troubleshooting. The values are positional and case-sensitive.

s: the server-side timeout expired while waiting for the server to send or receive data.

...so, timeout server fired, while...

H: the proxy was waiting for complete, valid response HEADERS from the server (HTTP only).

The server had not finished (perhaps not even started) returing all the response headers to the proxy, but the connection was established and the request had been sent.

HAProxy returns 504 Gateway Timeout, indicating that the backend did not respond in a timely fashion.

If your backend needs longer than 100 seconds (?!) then you need to increase timeout server. Otherwise, your Apache server seems to have a problem being too slow to respond.

Answer 2

I had a similar issue and found the problem was with how I had configured my backend server section.

backend no_match_backend
  mode http
  balance roundrobin
  option forwardfor
  option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
  server nginx-example 192.168.0.10 check port 80

My problem is that I did not specify the port for the connection. When connecting via HTTP it would work but as I have my SSL terminated on my haproxy. This attempts to connect via 443 to the backends. As the backends cannot / don't correctly communicate. The setup of the SSL session with haproxy and the backend that causes the gateway to time out. I need to force unencrypted communications to the backends.

backend no_match_backend
  mode http
  balance roundrobin
  option forwardfor
  option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
  server nginx-example 192.168.0.10:80 check port 80

The change might be hard to spot server nginx-example 192.168.0.10 check port 80 now has :80 after the ip 192.168.0.10:80

This problem was made more complicated by my backend servers having SSL redirects configured. So all my requests would arrive as HTTP and be redirected to HTTPS. So it was difficult to identify where the problem was. I It looked like https requests were being redirected correctly to the backend servers. I need to disable this redirect on the backend servers and move it forward to haproxy config.