How to terminate AWS EMR Cluster automatically after some time

Question

I currently have a task at hand to Terminate a long-running EMR cluster after a set period of time (based on some metric). Google Dataproc has this capability in something called "Cluster Scheduled Deletion" Listed here: Cluster Scheduled Deletion

Is this something that is possible on EMR natively? Maybe using Cloudwatch metrics? Or can I write a long-running jar which will sit on the EMR Master node and just poll yarn for some idle time metric and then shut down the cluster after a set period of time?

Edit: For more clarification. I would like some functionality wherein the cluster is terminated based on idle for some x amount of time. e.g. If the cluster has been up for a while but no jobs have been run for say 1 hour and the cluster is just sitting there doing nothing, then I'd like the ability to terminate the cluster.

Answer 1

The easiest method would be used to Amazon EMR Metrics and Dimensions for Amazon CloudWatch. There is an isIdle boolean that "indicates that a cluster is no longer performing work".

You could create a CloudWatch Alarm that says if it is True for more than x minutes, then trigger the alarm. This would send a message to Amazon SNS, which can trigger a Lambda function to shutdown the cluster.

Components:

• Amazon CloudWatch Alarm
• Amazon SNS queue
• AWS Lambda function

Update: This apparently isn't suitable (see comments below).

An alternate method would be:

• Use Amazon CloudWatch Events to schedule a Lambda function every x seconds
• The Lambda function looks for any clusters with a particular tag that indicates how long to wait until shutdown (eg 40 minutes). If the tag is not present, the cluster remains untouched.
• The Lambda function queries the cluster state (somehow -- probably via a Hadoop API call), then:
  • If the cluster is idle and there is no Idle Since tag, add an Idle Since tag with the current timestamp
  • If the cluster is idle and it been more than x minutes since the timestamp in the Idle Since tag, terminate the cluster.
  • If the cluster is not idle, remove the Idle Since tag (if present)

Answer 2

Keeping in mind the clarification that you have provided in your question, there could be 3 possible ways to do that.

1) Using AWS CloudWatch metric isIdle of an EMR cluster. This metric tracks whether a cluster is live, but not currently running tasks. You can set an alarm to fire when the cluster has been idle for a given period of time, such as thirty minutes. Reference: https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_ViewingMetrics.html

2) [Recommended] Using AWS CloudWatch event/rule and AWS Lambda function to check for Idle EMR clusters. You can achieve visibility on the AWS Console level and can easily enable and disable it.

[Recommended] Solution using 2nd Approach

Keeping in mind the need for this, I have developed a small framework to achieve that using the 2nd solution mentioned above. This framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.

You specify the maximum idle time threshold and AWS CloudWatch event/rule triggers an AWS Lambda function that queries all AWS EMR clusters in WAITING state and for each, compares the current time with AWS EMR cluster's ready time in case of no EMR steps added so far or compares the current time with AWS EMR cluster's last step's end time. If the threshold has been compromised, the AWS EMR will be terminated after removing termination protection if enabled. If not, it will skip that AWS EMR cluster.

AWS CloudWatch event/rule will decide how often AWS Lambda function should check for idle AWS EMR clusters.

You can disable the AWS CloudWatch event/rule at any time to disable this framework in a single click without deleting its AWS CloudFormation stack.

AWS Lambda function is using Python 3.7 as its runtime environment.

You can get the code and use it from GitHub here: https://github.com/abdullahkhawer/auto-terminate-idle-emr

Note: Any contributions, improvements, and suggestions to this solution that I developed will be highly appreciated.

3) Some other custom solution based on a Shell that runs against a CRON job on an EMR cluster's master node but you will lose its visibility on the AWS Console level and you may require SSH access as well.