EC2 Automatically Install Updates

Question

I've recently joined a company as a DevOps Engineer and one of my responsibilities is provisioning infrastructure in OpsWorks (EC2 instances behind an ELB talking to an Aurora DB).

I've been asked to audit the current solution with respect to applying updates.

After reading a few answers it strikes me that if I want to apply security updates to our (Centos) EC2 instances automatically then the options are:

1. Lambda job to spin up a new instance (which will run yum update), attach that to the ELB and then terminate the 'old' instance
2. Add a cron job to each instance (there are about a dozen) which runs yum update --security every day or so
3. Add custom JSON to the OpsWorks layer such that the yum update runs after each deployment

There are a number of other ways to do this each with pros and cons so I'd like to know what the consensus is on 'best practice' for automatically updating several instances on a regular basis.

NOTE: These instances are running 24/7 365 (i.e. we never restart them), and code deployments can range from daily to monthly depending on the instance!

Answer 1

OK, there will be lots of opinions on this one.

Do not automatically install updates, even security ones. One day you will find all your systems down and you will be scrambling to figure out why. Then management will be scrambling to figure out the cost of an enterprise system on cloud vendor XYZ having crashed.

No updates, no matter how big or small, security or application patch, should be installed blindly. You should have test instances where you install patches and updates and verify the impact on the system, your applications, etc. These steps should be documented as a devops procedure.

Only after having verified that the patch or update performs as expected, do you roll out your instance update strategy (rolling, blue/green, etc.).

Some people will argue that this means you will wait too long to install a critical security patch. I argue that having good devops procedures means that you are monitoring the patches and updates on a constant basis (like every morning). Depending on the severity of the bug you will then determine if an emergency update should occur or let standard OP (operating practice) continue.