When to provision in Packer vs Terraform?

Question

I am sitting with a situation where I need to provision EC2 instances with some packages on startup. There are a couple of (enterprise/corporate) constraints that exist:

• I need to provision on top of a specific AMI, which adds enterprisey stuff such as LDAP/AD access and so on
• These changes are intended to be used for all internal development machines

Because of mainly the second constraint, I was wondering where is the best place to place the provisioning. This is what I've come up with

Provision in Terraform

As it states, I simply provision in terraform for the necessary instances. If I package these resources into modules, then provisioning won't "leak out". The disadvantages

• I won't be able to add a different set of provisioning steps on top of the module?
• A change in the provisioning will probably result in instances being destroyed on apply?
• Provisioning takes a long time because of the packages it tries to install

Provisioning in Packer

This is based on the assumption that Packer allows you to provision on top of AMIs so that AMIs can be "extended". Also, this will only be used in AWS so it won't use other builders necessarily. Provisioning in Packer makes the Terraform Code much simpler and terraform applies will become faster because it's just an AMI that you fire up.

For me both of these methods have their place. But what I really want to know is when do you choose Packer Provisioning over Terraform Provisioning?

Answer 1

Using Packer to create finished (or very nearly finished) images drastically shortens the time it takes to deploy new instances and also allows you to use autoscaling groups.

If you have Terraform run a provisioner such as Chef or Ansible on every EC2 instance creation you add a chunk of time for the provisioner to run at the time you need to deploy new instances. In my opinion it's much better to do the configuration up front and ahead of time using Packer to bake as much as possible into the AMI and then use user data scripts/tools like Consul-Template to provide environment specific differences.

Packer certainly can build on top of images and in fact requires a source_ami to be specified. I'd strongly recommend tagging your AMIs in a way that allows you to use source_ami_filter in Packer and Terraform's aws_ami data source so when you make changes to your AMIs Packer and Terraform will automatically pull those in to be built on top of or deployed at the next opportunity.

I personally bake a reasonably lightweight "Base" AMI that does some basic hardening and sets up monitoring and logging that I want for all instances that are deployed and also makes sure that Packer encrypts the root volume of the AMI. All other images are then built off the latest "Base" AMI and don't have to worry about making sure those things are installed/configured or worry about encrypting the root volume.

By baking your configuration into the AMI you are also able to move towards the immutable infrastructure model which has some major benefits in that you know that you can always throw away an instance that is having issues and very quickly replace it with a new one. Depending on your maturity level you could even remove access to the instances so that it's no longer possible to change anything on the instance once it has been deployed which, in my experience, is a major factor in operational issues.

Very occasionally you might come across something that makes it very difficult to bake an AMI for and in those cases you might choose to run your provisioning scripts in a Terraform provisioner when it is being created. Sometimes it's simply easier to move an existing process over to using provisioners with Terraform than baking the AMIs but I would push to move things over to Packer where possible.

Answer 2

I have come across the same situation. As per my understanding

• If you bring up your EC2 instances very frequently say 2 to 3 times a day then go with creating an customized AMI with packer and then call the ami through terraform.
• If your base image ( AMI created by packer) change frequently based on your requirements then its good to go with packer. But for me running a packer scripts are very time consuming.
• You can do the same with packer as well. You can write your requirements in a script and call it in terraform. By having everything incorporated in a terraform script would reduce some time

Finally its your decision and your frequency of bringing up EC2 instances.