What does terraform refresh really do?

Question

While using terraform to deploy a fairly large infrastructure in AWS, our remote tfstate got corrupted and was deleted.

From the documentation, I gather that terraform refresh should query AWS to get the real state of the infrastructure and update the tfstate accordigly, but that does not happen: my tfstate is untouched and plan + apply give a lot of Already existing errors.

What does terraform refresh really do?

Answer 1

terraform refresh attempts to find any resources held in the state file and update with any drift that has happened in the provider outside of Terraform since it was last ran.

For example, lets say your state file contains 3 EC2 instances with instance ids of i-abc123, i-abc124, i-abc125 and then you delete i-abc124 outside of Terraform. After running terraform refresh, a plan would show that it needs to create the second instance while a destroy plan would show that it only needs to destroy the first and third instances (and not fail to destroy the missing second instance).

Terraform makes a very specific decision to not interfere with things that aren't being managed by Terraform. That means if the resource doesn't exist in its state file then it absolutely will not touch it in any way. This enables you to run Terraform alongside other tools as well as making manual changes in the AWS console. It also means that you can run Terraform in different contexts simply by providing a different state file to use, allowing you to split your infrastructure up into multiple state files and save yourself from catastrophic state file corruption.

To get yourself out of your current hole I suggest you use terraform import liberally to get things back into your state file or, if possible, manually destroy everything outside of Terraform and start from scratch.

In future I would suggest both splitting out state files to apply for more granular contexts and also to store your remote state in an S3 bucket with versioning enabled. You could also look towards tools like Terragrunt to lock your state file to help avoid corruption or wait for the native state file locking in the upcoming 0.9 release of Terraform.