How to keep node app running?

Question

I've finally finished converting one of my projects to use Node.JS, but now I'm having problems keeping my app running on the server :/ The problem is that if I close my putty session Node just stops.

I've done a lot of searching on this problem, and it seems that creating an upstart script and using the Forever module is the way to go.

I started googling and created this upstart script:

#!upstart
description "Loner NodeJS app launcher"
author      "[email protected]"

start on startup
stop on shutdown

script
    export HOME="/root"
    exec sudo node /home/jjmpsp/server.js >> /home/jjmpsp/server.sys.log 2>&1
end script

I then ran start app on the server last night and the server stayed running when I closed the putty session. All good.

However, I came on this morning and discovered that the Node app had stopped so I checked the server.sys.log file to see what was going on. It seems that the app ran fine until it eventually encountered this exception:

debug: client authorized
info: handshake authorized fziLHZA3Vo9i55eubvOq

events.js:48
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: Connection lost: The server closed the connection.
    at Protocol.end (/home/jjmpsp/node_modules/mysql/lib/protocol/Protocol.js:73:13)
    at Socket.onend (stream.js:80:10)
    at Socket.emit (events.js:88:20)
    at TCP.onread (net.js:348:51)

Today I've been googling even more and discovered that Forever will actually restart a NodeJS app if it exits unexpectedly. I tried installing the module with npm install forever but I get this huge list of errors:

jjmpsp@alex:~$ npm install forever
npm ERR! error installing [email protected] Error: No compatible version found: node-fork@'>=0.4.0- <0.5.0-'
npm ERR! error installing [email protected] No valid targets found.
npm ERR! error installing [email protected] Perhaps not compatible with your version of node?
npm ERR! error installing [email protected]     at installTargetsError (/usr/local/lib/node_modules/npm/lib/cache.js:424:10)
npm ERR! error installing [email protected]     at /usr/local/lib/node_modules/npm/lib/cache.js:406:17
npm ERR! error installing [email protected]     at saved (/usr/local/lib/node_modules/npm/lib/utils/npm-registry-client/get.js:136:7)
npm ERR! error installing [email protected]     at Object.cb [as oncomplete] (/usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:36:9)
npm ERR! Error: No compatible version found: node-fork@'>=0.4.0- <0.5.0-'
npm ERR! No valid targets found.
npm ERR! Perhaps not compatible with your version of node?
npm ERR!     at installTargetsError (/usr/local/lib/node_modules/npm/lib/cache.js:424:10)
npm ERR!     at /usr/local/lib/node_modules/npm/lib/cache.js:406:17
npm ERR!     at saved (/usr/local/lib/node_modules/npm/lib/utils/npm-registry-client/get.js:136:7)
npm ERR!     at Object.cb [as oncomplete] (/usr/local/lib/node_modules/npm/node_modules/graceful-fs/graceful-fs.js:36:9)
npm ERR! Report this *entire* log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[email protected]>
npm ERR!
npm ERR! System Linux 3.8.4-x86_64-linode31
npm ERR! command "node" "/usr/local/bin/npm" "install" "forever"
npm ERR! cwd /home/jjmpsp
npm ERR! node -v v0.5.11-pre
npm ERR! npm -v 1.0.106
npm ERR! Error: EACCESS, Permission denied 'npm-debug.log'
npm ERR! Report this *entire* log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[email protected]>
npm ERR!
npm ERR! System Linux 3.8.4-x86_64-linode31
npm ERR! command "node" "/usr/local/bin/npm" "install" "forever"
npm ERR! cwd /home/jjmpsp
npm ERR! node -v v0.5.11-pre
npm ERR! npm -v 1.0.106
npm ERR! path npm-debug.log
npm ERR! code EACCESS
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/jjmpsp/npm-debug.log
npm not ok

What steps should I take to fix this? I'm completely out of ideas. I've been googling all sorts of technical details and I'm just not getting anywhere.

Any help is greatly appreciated :)

Answer 1

1. First, you should focus on what is killing your node server. Forever isn't going to "fix" the problem. Every time node exits/restarts it will cause problems and your users can loose data. upstart or forever are just band-aids. (In fact, Upstart will restart your server, but it will give up if your server doesn't stay running.)

   The only long-term solution to find/fix each source of errors and write a regression test suite to verify that each previous problems has been fixed.

2. start on startup will not work

3. Your forever install broke because of permissions. Try sudo npm install -g forever Advanced: Your entire server setup should be scripted. That way you can set up a test/staging server that mirrors production. one simple way to do this is Vagrant. You can develop "locally" on the same OS that's on your server. And you can test things like "does node come up when I reboot?" or "if the server gets wiped out, can I re-create my server from the base OS?"

Answer 2

After a painful few hours of re configuring everything, I think I have finally solved my problem! It looks like I may have had 2 versions of node or something! For future reference: If you are new to Node, be sure to install nvm to make managing node versions easier and you won't experience this problem :)

Answer 3

After a lot of research and bad solutions, I found pm2 (part of Keymetrics, recommended by Karlos) to be by far the best solution. You can find it here:

http://pm2.keymetrics.io/

Answer 4

I think forever doesn't exits your nodejs app unexpectedly, It might be because of the mysql 'wait_timeout' parameter which is set to 28800 second (i.e 8 hours) by default. If the nodejs app does not interact with mysql database for 8 hour, then the connection gets closed. You have to set the wait_timeout value to 'MAX value'(i.e 365 days) or any other value as you required in the mysql db settings.

Link for the wait_timeout settings:

http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_wait_timeout

or

Add the following code in your code. This code will ping the mysql db once in an hour and your connection will not break.

// MYSQL ping code starts

function keepalive() 
{
     conn.query('select 1', [], function(err, result) 
     {
         if(err) 
         return console.log(err);
         // Successul keepalive
         console.log("keepalive: "+ result);
     });
}

setInterval(keepalive, 1000*60*60);

// MYSQL ping code ends

You can also use the handledisconnect() code and handle the particular connection error in the code as given in the example below.

///Code starts

function handledisconnect() {

  log.info("handledisconnect()");
    // Reading config.json file for database connection
    var db_config = JSON.parse(fs.readFileSync(appRoot + "/conf.json"));

    conn = mysql.createConnection(db_config);

    conn.connect(function(err) {

        if (err) {

            log.error('error when connecting to db : ',err);
            // We introduce a delay before attempting to reconnect
            setTimeout(handledisconnect, 200);
        } else {
            log.info("database connected");
        }
    });

    conn.on('error', function(err) {

        log.error('db error : ',err);

        // handling the reconnection for 'PROTOCOL_CONNECTION_LOST' error
        if (err.code === 'PROTOCOL_CONNECTION_LOST') {

            log.error(err);
            setTimeout(handledisconnect, 200);
            handledisconnect();

        } else {
            log.error(err);
        }
        // handling the reconnection for 'EADDRINUSE' error
        if (err.code === 'EADDRINUSE') {

            setTimeout(handledisconnect, 200);
            handledisconnect();

        } else {

            log.error(err);
        }
        // handling the reconnection for 'ECONNREFUSED' error
        if (err.code === 'ECONNREFUSED') {

            log.error(err);
            setTimeout(handledisconnect, 200);
            handledisconnect();

        } else {
            log.error(err);
        }

    });

///**** ADD YOUR CODE HERE****////

}
handledisconnect();
/// Code ends