Intermittent 502 bad gateway error(using nginx,nodejs,mongodb)

Question

We are using nodejs(v 0.10.29 ) ,express,nginx( version 1.4.6) with mongodb(v 2.6.3) replicaset and getting intermittent 502 bad gateway error. pm2 logs is unable to log error though nginx aerror.log is showing

recv() failed (104: Connection reset by peer) while reading response header from     upstream, client: xxx.xxx.xxx.xxx, server: somedomain.com, request: "GET /img/abc.png HTTP/1.1", upstream: "http://127.0.0.1:3000/img/abc.png", host: "domain.com", referrer: "http://domain.com/admin/"

and access.log is saying:

"GET /url/abc.html HTTP/1.1" 502 723 "http://domain.com/admin/" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36"

can anyone guide me with the issue?

Answer 1

In our case nginx was on top of Node.js, which was automatically started by Forever. Due to an error in Redis database contents the Node server crashed intermittently and nginx returned either a 502 or 503 error. It took us some time to find the reason, as Node's crashing messages were logged only by Forever.

So, no failure on nginx or its configurations, the Node server (and its background services) were the source.

Answer 2

It probably isn't a problem with nginx per se but more of a problem with nodejs. The 502 bad gateway error means that nginx asked the nodejs server for some information, and the nodejs server answered and then "promptly hung up the phone." The "hung up the phone" is more of a reference to the "Connection reset by peer" part of the message.

Now, this could be indicative of many different problems, so I can't give you a definitive answer.

It could be that there was an actual error in processing the request, which would require tracing the bug through the js code.

It could be a memory issue (what does your memory usage look like)?

Or it could be a timeout error on node's part, either because it took too long to return an answer or because the code was buggy or because the system ran out of memory.

I know that I used to get intermittent timeout errors between nginx and php-fpm because nginx would give up waiting for php. I adjusted the memory usage and the timeout settings to fix that (and I optimized the php code).

If you can provide more specific information about memory / load usage or logs from node or the application or even a general pattern in the 502 errors (is it localized to a geographic region, a browser, an OS?), then the answer could be less speculative.