Why am I getting an Apache Proxy 503 error?

Question

My server was doing just fine up until yesterday. It was running Redmine, and it was the happiest little server until my "friend" imported a SQL table that my little guy couldn't take. Unfortunately, after an hour of trying to get the lil guy to respond, we had to power cycle him.

Now after restart, we get a 503 error when trying to visit the domain connected to Redmine. It's hooked up to a Mongrel daemon, and we use Apache Proxy to direct all connections to the port Redmine is running on.

Using Lynx on the server (http://localhost:8000) you can see the Ruby application working fine. But this bit is not working in my Apache configuration file:

<VirtualHost *:80>     ServerName sub.example.com     ProxyPass / http://localhost:8000     ProxyPassReverse / http://localhost:8000     ProxyPreserveHost on     LogLevel debug </VirtualHost> 

Here's the error log output for Apache:

 [debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000 [debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/ [debug] mod_proxy.c(756): Running scheme http handler (attempt 0) [debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/ [debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost) [debug] proxy_util.c(1815): proxy: connecting http://localhost:8000/ to localhost:8000 [debug] proxy_util.c(1908): proxy: connected / to localhost:8000 [debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket created to connect to localhost  [error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed [error] ap_proxy_connect_backend disabling worker for (localhost)  [debug] proxy_util.c(1773): proxy: HTTP: has released connection for (localhost) 

Answer 1

Apache will respond with 503's for at least 60 seconds any time it detects that the backend server is down. This is the default behavior. As in your example, if you restart your backend server (Rails in this example) and someone tries to access it through the Apache proxy before Rails is ready then Apache will return 503's for the next 60 seconds regardless if your backend is now 'up'. Please see the apache docs on ProxyPass where it states:

retry 60

Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance, and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.

So if you set your Proxy Pass to include retry=0 you won't see the 503's when you restart your backend service. This is also useful when using Apache as reverse proxy during development! For example:

ProxyPass / http://localhost:8000 retry=0

Answer 2

Run following command

# /usr/sbin/setsebool httpd_can_network_connect 1 

OR

# /usr/sbin/setsebool httpd_can_network_connect true 

and after that restart httpd

# service httpd restart