Post hook for Elastic MapReduce

Question

I wonder if there is an example of post process for EMR (Elastic MapReduce)? What I am trying to achieve is send an email to group of people right after Amazon's Hadoop finished the job.

Answer 1

You'll want to configure the job end notification URL.

jobEnd.notificationUrl

AWS will hit this url, presumably with query variables that indicate which job has completed (job id).

You could then have this URL on your server process your email notifications, assuming you had already stored a relationship between emails and job ids.

https://issues.apache.org/jira/browse/HADOOP-1111

Answer 2

An easier way is to use Amazon CloudWatch (monitoring system) and Amazon Simple Notification Service (SNS) to monitor and notify you and others on the status of your EMR jobs.

For example you can set an alarm for your cluster to check when it IsIdle. It will be set to 1 once the job is done (or failed), and you can then get SNS notification as an email (or SMS even). You can set similar alarms on count of JobsFailed and other metrics.

For the complete list of EMR related metrics you can see EMR documentations

You can see more information about it here: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_ViewingMetrics.html