persist logs of unhealthy instances in elastic beanstalk before termination

Question

I've just had a situation where my ebs app detected an un-healthy instance over the weekend and in response swapped the un-healthy instance for a new one.

That's fine and what I want to happen but I've realised the log files that would tell me why the instance became unhealthy have been erased with the unhealthy instance.

What needs to be put into place to say save the logs files of an unhealthy instance to s3 before termination? Is there a setting I need to enable, or do I have to write some code myself to listen to the ebs events and save the logs to s3 (would have thought this is quite a common requirement)?

Thanks

Answer 1

Ok so after searching around there doesn't seem to be a quick setting you can enable on ec2/beanstalk to save the logs (which is a bit disappointing).

There is the option of using a third party logging service such as papertrail or loggly but I think that is overkill for what I need.

So after doing more digging I've finally managed to implement what I need by adding a script to /etc/init.d on the ec2 box. I'll post exactly what I did below but first just want to give credit to Arun Kumar series of posts on hudku.com from which I borrowed from a lot and akhtet6 on the aws forum who explained how to get the init.d script to run on machine shutdown.

Basically you need to create a folder in the root of your app's war file called '.ebextensions' and then place all of the following files inside that folder.

elastic-beanstalk.config (this is a YAML file and one of the file I borrowed from Arun Kumar's posts)

# Errors get logged to /var/log/cfn-init.log. See Also /var/log/eb-tools.log

container_commands:
    01-command:
        command:        rm -rf /custom/.ebextensions

    02-command:
        command:        mkdir -p /custom/.ebextensions

    03-command:
        command:        cp -R .ebextensions/* /custom/.ebextensions/

    04-command:
        command:        chmod 700 /custom/.ebextensions/app-setup.sh

    05-command:
        command:        bash /custom/.ebextensions/app-setup.sh

app-setup.sh (this is the other file I borrowed from Arun Kumar's posts and modified for my purposes)

#!/bin/bash

# Set DEBUG to 1 to debug this script. 2 for debugging scripts called by this script and so on.
# Execute "export DEBUG=1" to debug this script.
# Set value to 2 to debug this script and the scripts called within this script.
# Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged.
[[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1))

# Check if this is the very first time that this script is running
if ([ ! -f /root/.not-a-new-instance.txt ]) then
    newEC2Instance=true
fi

# Get the directory of 'this' script
dirCurScript=$(dirname "${BASH_SOURCE[0]}")

# Redirect stdout and stderr and append it to our file
curDateTime=$(date "+%Y%m%d%H%M%S")
exec &>> /usr/share/tomcat7/logs/customise-setup-log-$curDateTime.txt
echo $(date)

echo "Setting up app"
pwd

# Set permissions
chmod 777 /custom/.ebextensions/*
ls -al /custom/.ebextensions

# Set-up saveLogsToS3 service in appropriate run levels
cp /custom/.ebextensions/saveLogsToS3 /etc/init.d/
touch /var/lock/subsys/saveLogsToS3

/sbin/chkconfig saveLogsToS3 --level 12345 on
/sbin/chkconfig saveLogsToS3 --level 06 off


# If new instance, now it is not new anymore
if ([ $newEC2Instance ]) then
    echo -n "" > /root/.not-a-new-instance.txt
fi

# Print the finish time of this script
echo $(date)

# Always successful exit so that beanstalk does not stop creating the environment
exit 0

saveLogsToS3 this is the init.d script

#!/bin/sh
#
# chkconfig: 0123456 99 01
# description: 
#

RETVAL=0

start () {
    touch /custom/.ebextensions/saveLogsToS3_START
    touch /var/lock/subsys/saveLogsToS3
}

stop () {
    touch /custom/.ebextensions/saveLogsToS3_STOP
    /custom/.ebextensions/copylogs.sh > /custom/.ebextensions/saveLogsToS3_STOP.log
    RETVAL=$?
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        start
        ;;
    *)
        start
        ;;

esac
exit $RETVAL

copylogs.sh (this is the file that performs the copy, replace all angled bracketed vars with appropriate values)

#!/bin/bash

output=$(/opt/aws/bin/ec2-metadata -i)
instance=${output:13}

s3put --region eu-west-1 -a <Access_Key> -s <Secret_Key> -b <s3_bucket> -p /usr/share/tomcat7/logs/ -k Terminated_logs/$instance put /usr/share/tomcat7/logs/*

Once you deploy your new app version you can test that it works by ssh'ing on to the ec2 box and running 'sudo reboot' and checking your s3 bucket for the log files.