502 Bad Gateway... invalid response from upstream server (apache and jboss)

Question

I'm using apache on one machine to serve requests to jboss on another machine. Everything works fine when I startup jboss and I'm able to access the web app, but after a few hours I eventually start getting "The proxy server received an invalid response from an upstream server" errors. If I restart jboss then everything works fine again, but several hours later I have the same issue...

Does anybody know what could be causing this issue? I don't have access to the apache logs at this time (I should in a few hours) but it seems to be something with jboss since restarting it is the temporary fix.

I'm using jboss4.2.3 and apache 1.3 with mod_jk. I'm not finding any errors in the jboss logs and the app I'm trying to reach isn't doing anything that takes a long time. The main page is just a simple login page. I have ports 8009 and 8080 open for communication between the app server and web server. I don't know what configuration is wrong.

Answer 1

This sounds to me like mod_jk in Apache is getting out of sync with the AJP connector in JBoss. The AJP protocol uses persistent, re-used connections between web server and app server, and if the protocol is not configured exactly the same on both ends of the connection, eventually the connections go stale at one end of the connection, but the other end keeps trying to use them. The symptom is a 502 error.

My first suggestion is this: don't use mod_jk unless you need to. It's complex and hard to configure to get a stable system. If you don't need its performance or load balancing features, I suggest using mod_proxy instead. It's just as good for most applications, and pretty easy.

But if you want to stick to mod_jk, The first thing you need to is make sure you're using the very latest mod_jk version (currently 1.2.28), since older versions are notoriously hard to configure. Luckily, mod_jk is still supported on Apache 1.3.

Next, check the mod_jk log file (configured using the JkLogFile directive). If you're seeing a bunch of connection-related errors around the time things go wrong, you need to tweak your jk config at both ends of the connection. The most likely culprit is the timeout settings, so read up about those here, and make sure both ends are singing from the same hymn sheet.

Answer 2

I've also seen this occur using apache and tomcat. In my particular situation, the application deployed to tomcat had a bug that caused response threads to hang. Eventually tomcat ran out of worker threads, and apache wasn't able to establish a connection.

In our case, database connections weren't getting released properly back into a connection pool, and other threads were waiting indefinitely to obtain a connection from the pool. However, anything that indefinitely keeps alive a response handling thread could lead to the same problem.