Node JS auto restart all forever JS process when server goes down / crashes

Question

I am using forever js to keep my node server running 24/7 on AWS EC2.

I use this command

forever start index.js

However, I notice that some time it randomly stops the process and my site goes down. I have to manually ssh into my server to run it again by doing:

forever restartall

And then it goes backup. Is there any way by which I can define a timeout, lets say if the server/website does not respond for 200 in 5 sec, then restart all forever process automatically?

I am new to this, if any one can give me step by step example for my case, it would be awesome.

Answer 1

A (NodeJS) server should not stop for no reason. Most of the time, it's because of a 500 Error that have not been catched and stop the server, then you will have to restart it. forever is using node by default to start your server.

nodemon is a npm package that restart your server when the code changes or when your server stops.

You can use forever and nodemon together by doing :

forever start nodemon --exitcrash app.js 

or

forever start -c nodemon app.js 

Or, as suggested in other answers, you can use PM2, which would be better for production !

Answer 2

I'd suggest the usage of PM2

This is the best option to run on a production server.

What are the advantages of running your application this way?

• It's easy to setup and run.
• PM2 will automatically restart your application if it crashes.
• PM2 will keep a log of your unhandled exceptions - in this case, in a file at /home/safeuser/.pm2/logs/app-err.log.
• With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically meaning your node application will start as a service.

ref: https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps

Answer 3

So this is an example of using cronto run scripts that can restart service/perform some automated task. Basically, I created some scripts that I need to run at certain time intervals on my server. For your case, you want to make a script that will automatically check the state of your forever.js and if it returns a bad response, run the forever restartall command that you mention above.

You can set this up by creating a new crontab entry on the server. As far as the script goes, I'm by no means a bash script guru; I made a simple script that works for me. Here is an example of checking a service on my machine, restarting it if it is not running.

#!/bin/bash
zabbix_server="service zabbix-server"
zabbix_agent="service zabbix-agent"
logfile=zabbix_auto_restart.log
logfilePath=/etc/scripts/zabbix/$logfile
zabbix_server_running=0
zabbix_agent_running=0

grep_agent (){
        local retval=$(ps -ef | grep -v grep | grep zabbix_agentd | wc -l)
        echo $retval
}

grep_server (){
        local retval=$(ps -ef | grep -v grep | grep zabbix_server | wc -l)
        echo $retval
}

check_zabbix_agentd (){
        if (( $(grep_agent) <= 0 ))
        then
           sudo /etc/init.d/zabbix-agent start
           echo `date` "$zabbix_agent was stopped... Restarting" >> $logfilePath
           echo "************************************************" >> $logfilePath

           #Send email to notify that the script ran
           echo "$(date) $zabbix_agent was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" <my-email>

        else
           let zabbix_agent_running=1
        fi
}

check_zabbix_server (){
        if (( $(grep_server) <= 0 ))
        then
           sudo /etc/init.d/zabbix-server start
           echo `date` "$zabbix_server was stopped... Restarting" >> $logfilePath
           echo "************************************************" >> $logfilePath

           #Send email to notify that the script ran
           echo "$(date) $zabbix_server was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" [email protected]

        else
           let zabbix_server_running=1
        fi
}

main_loop (){
        until ((zabbix_server_running == 1 && zabbix_agent_running == 1));
        do
                check_zabbix_agentd
                check_zabbix_server
                sleep 1.5
        done
}

main_loop