Node stops responding after a while

Question

My node server is running normally. Working with absolutely no issues. Then at a certain (random?) point in time, it stops responding to requests. (It did respond for hours uninterrupted before.)

There are no errors. The server is still able to do internal processes, but is not responding to requests.

With that, the requests are making their way to the server as the log shows:

GET / - - ms - -
GET / - - ms - -
GET /upload - - ms - -
GET /profile - - ms - -

I can't figure out when it is triggered, what is happening, and why it's happening.

Once the server stopped responding, it will never respond again. Restarting the server solves the issue until it happens again.

Anybody ever encountered this behavior?

Update:

1. The server never crashes, just stops responding.
2. This app is over a year old, and was working fine, so I doubt it's a basic, but rather something else.
3. Recent change: Started using YouTube SPF
4. Recent change: Connecting to a MongoDB on a remote server rather than localhost

Again, the server never crashes, just stop responding.

These are the components I am using for the server:

var express = require('express');
var app = express();
var port = process.env.PORT || 8080;
var morgan = require('morgan');
var cookieParse = require('cookie-parser');
var session = require('express-session');
var mongoose = require ('mongoose');
var bodyParser = require('body-parser');
var configDB = require('./config/database');
var passport = require('passport');
var flash = require('connect-flash');
var MongoStore = require('connect-mongo')(session);
var https = require("https");
var path = require('path');
var helmet = require ("helmet");
var robots = require("express-robots");
var Sitemap = require('sitemap')
var async = require('async')
var cron = require('node-schedule');
var compression = require('compression')

Update:

Added annotated pm2 logs:

0|server   | GET /public/js/fullscreen.js 304 72.626 ms - -
0|server   | GET /public/js/mathUtils.js 304 72.291 ms - -
0|server   | GET /public/js/snap.svg.js 304 74.047 ms - -
0|server   | GET /views/58dc059bf5ac4394447e7a4d/82z7rw8qbvgqfr 304 95.918 ms - -
0|server   | GET /public/js/playerTransport.js 304 96.425 ms - -
0|server   | GET /public/res/img/rightarrow.png 304 82.923 ms - -
0|server   | GET /public/res/img/BecomicsLogo.png 304 93.323 ms - -
0|server   | GET /public/res/img/leftarrow.png 304 84.158 ms - -
0|server   | GET /public/res/img/fullscreen.png 304 93.776 ms - -
0|server   | GET /public/fonts/foundation-icons.woff 304 61.611 ms - -
0|server   | GET /public/images/ui-bg_glass_100_f6f6f6_1x400.png 404 57.253 ms - 39
0|server   | GET /public/images/ui-icons_ef8c08_256x240.png 404 57.646 ms - 39
0|server   | POST /rating/get 200 111.758 ms - 37
0|server   | POST /player/getBook?id=58dc059bf5ac4394447e7a4d 200 99.961 ms - -
0|server   | GET /public/res/img/loading.gif 304 70.614 ms - -
0|server   | POST /stats/pageView 200 73.545 ms - 2
0|server   | GET /subscriptions 304 655.215 ms - -
0|server   | GET /public/css/discover.css 304 72.171 ms - -
0|server   | GET /public/css/becomics.css 304 73.344 ms - -
0|server   | GET /public/js/spf.js 304 65.621 ms - -
0|server   | GET /public/res/img/hamburgerMenu.png 304 58.525 ms - -


0|server   | POST /subscriptions/count 200 111.033 ms - 11 <--- This is the last time the server will respond
0|server   | GET /profile/5825063e47234fc77cbc7694/$otherUserPic - - ms - -
0|server   | GET /profile/5825063e47234fc77cbc7694/$picTheWhat - - ms - -


0|server   | So far so good <--This prints just before the res.send(). It verifies that the request makes it all the way to the response. But still. No response... 
0|server   | So far so good
0|server   | GET /profile/5825063e47234fc77cbc7694/$otherUserPic - - ms - -
0|server   | GET /profile/5825063e47234fc77cbc7694/$picTheWhat - - ms - -
0|server   | GET /subscriptions - - ms - -
0|server   | GET /profile/5825063e47234fc77cbc7694 - - ms - -
0|server   | GET /profile/5825063e47234fc77cbc7694 - - ms - -

0|server   | Counting Views. <--- this is a croned operation. It executes with no problem. Even after server stopped responding.
0|server   | GET /profile/5825063e47234fc77cbc7694 - - ms - -
0|server   | GET / - - ms - -
0|server   | GET /profile/5806a8116b9b49b4201f5487 - - ms - -
0|server   | GET /trendy?spf=navigate - - ms - -
0|server   | GET /trendy?spf=navigate - - ms - -
0|server   | GET /fresh?spf=navigate - - ms - -
0|server   | GET /series/58dc032ef5ac4394447e7a3c - - ms - -
0|server   | GET / - - ms - -
0|server   | GET /featured?spf=navigate - - ms - -
0|server   | GET / - - ms - -
0|server   | GET /featured - - ms - -
0|server   | GET / - - ms - -
0|server   | GET /ping - - ms - -

As you can see. No errors. Any ideas?

Also note that /ping is a route that is just supposed to return status 200. No db querying, no nothing. The simplest response possible, and that fails too.

Update Tried using different browser + chrome's incognito to verify that it is not a cache issue. Still no response.

PM2 Monit:

Mem: 140MB CPU:0%
Loop delay 1.46ms

CPU Profile:

Heap snapshot

Update 4/2/2017: After some debugging I think it has to do with a disconnect from the remote server... Once I resolve this I will post answer.

Update 4/3/2017 Here is my mongooose connect code:

mongoose.connect(configDB.url,{user:configDB.user,pass:configDB.pwd,server:{reconnectTries:60,reconnectInterval:1000}});
// CONNECTION EVENTS
// When successfully connected
mongoose.connection.on('connected', function () {
    console.log('Mongoose default connection open to ' + configDB.url);
});

// If the connection throws an error
mongoose.connection.on('error',function (err) {
    console.log('Mongoose default connection error: ' + err);
    console.log("Disconnecting")
    mongoose.disconnect();
});
// When the connection is disconnected
mongoose.connection.on('disconnected', function () {
    console.log('Mongoose default connection disconnected');
    //It automatically reconnects with not issues
});

The thing is, non of these listeners fire when my problem occurs. No errors or exceptions thrown. No disconnect triggered. Just Halts..

Answer 1

It would be better if you could share your code so that we're more clear about the problem.

If you're using express, It could simply be the wrong ordering of app.use , or its possible that you have a faulty middleware. You can use a debugger or just find out on which requests your server crashes and give it a fix .