Node.js get notification for high memory usage

Question

I have a Node.js app running on Ubuntu 14 on Amazon EC2.

I want to send an email in case that the memory usage reaches a specific size.

I know that PM2 exposes an API that allows, among other things, restarting the app whenever a certain amount of memory usage is reached. Now I don't want to restart the app at this point, but just to get a notification about it, and doing with it whatever I want (in my case, sending an email).

How can I do it either with PM2 or any other free tool?

Answer 1

The answer is to use AWS CloudWatch alarms. They are free tier eligible and have a nice dashboard. Detailed setup is described inside this documentation guide, but I suggest you follow my steps to ensure it will work for you.

First thing you need to do is Launch a new Ubuntu EC2 instance that can write to CloudWatch . This involves a new IAM Role with permissions. (you can't attach a new Role to an existing instance - see second Note: here.). You no longer need to launch an EC2 instance to change IAM Roles. See announcement.

The next action you take is: Install the AWS authored perl scripts that allow you to write to CloudWatch. Add a new cron to write to CloudWatch every five minutes.

Lastly Create a new alarm in the CloudWatch console, to email you when the memory usage goes above a certain threshold.

---

Here are the steps for each phase listed above:

Install the AWS authored perl scripts

1. SSH into your new instance and run the following commands:

$ sudo apt-get update

$ sudo apt-get install unzip

$ sudo apt-get install libwww-perl libdatetime-perl

curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O

unzip CloudWatchMonitoringScripts-1.2.1.zip

rm CloudWatchMonitoringScripts-1.2.1.zip

cd aws-scripts-mon

2. Verify your perl scripts were installed correctly with the following command

./mon-put-instance-data.pl --mem-util --verify --verbose

3. Add to cron, the perl script command that puts the amount of memory utilized by the ubuntu instance into CloudWatch

crontab -e

*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

4. Wait about 20 minutes while statistics are added to CloudWatch.

---

Create a new alarm in the CloudWatch console

1. In the AWS console, select CoudWatch and click the Blue button, Browse Metrics.
2. In the right side of the screen you should see All Metrics tab in the bottom half of the screen and a link to Linux System. Click Linux System and then click InstanceId and you should see your MemoryUtilization metric.
3. Click MemoryUtilization and then switch to the Graphed Metrics tab in the bottom.
4. To the right here, you will see the Alarm icon.
5. Click this icon to create the alarm. Set the threshold to email you if the memory utilization goes above 40 as an example.
6. Add stress to the instance and you should see an email come in. I used the stress command found at this answer and it worked. Type stress and ubuntu will show you how to install stress. See below screen shot of my memory usage CloudWatch chart I generated for this write up. I got an email each time the memory usage crossed 40 percent.

Hope this helps.

Answer 2

To have a concrete implementation as matthewmatician mentioned.

You can easily do this all within node by periodically checking your memory usage: https://nodejs.org/api/process.html#process_process_memoryusage. If this is a long-running server, check it every hour or so. Then just fire off an email with a package like nodemailer if the memory reaches a certain threshold.

I like PM2, yet I was wondering if it could be done without.

As noted here, RSS (Resident Set Size) is the full memory usage of the process, this includes all memory usage of the shared memory. RSS can therefore be considered inacurate to be taken as the memory usage of a single process, since the shared memory is also used by other processes. Therefore PSS exists, which divides the shared memory over all other processes used.

I suppose we want to know the PSS value if we want to display the most accurate usage of memory by the node process. However, some person did mention PSS is more important for huge amount of fork processes, such as an Apache server (and thus a lot of shared memory).

var hour = 3600*1000;
var checkMemory = function() {
        // Retrieves the memory usage
        var memory = process.memoryUsage();
        var rss_memory_in_bytes = m.rss;
        var rss_memory_in_megabytes = m.rss / (1024**2);
        if (rss_memory_in_megabytes > 50) {
                // More than 50 megabytes RSS usage, do something
                doSomething();
        }
}
var doSomething = function() {
        // For instance sending an email
}
// Check the memory usage every hour
setInterval(checkMemory, hour);

-- UPDATED --

If there is more heap storage required, the Node process will attempt to allocate this memory. This allocation is done automatically. When successfull the heap storage increases and the rss as well. The heapUsage and heapTotal can therefore be neglected in our case.

There are ways of setting a memory limit, but we are interested at checking for a limit. I think it is reasonable to check for the amount of free system memory left. Yet, this has nothing to do with the actual memory usage of the Node process itself and would require a different threat on how to check for free system memory with a Node script.

Answer 3

You can a simple bash script that gets the memory usage of your instance and push that instance to the CloudWatch by using "custom metrics" feature. Then you can create alarms in Cloudwatch and make SNS sends email to you. ( You should create a cron job to run for example every 10 minutes ).

aws cloudwatch put-metric-data --metric-name memusage --namespace mem --value 20 --timestamp 2016-10-14T12:00:00.000Z