How should I handle `deposed` resources in Terraform?

Question

I am struggling to understand how I should handle deposed resources found in my TF state.

In particular, when running an apply, I am now getting some errors like the below:

Error applying plan:

11 error(s) occurred:

* aws_instance.mongo-replica-01 (deposed #0): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-0f0bdc2c16e922fbc' does not exist
    status code: 400, request id: 71b98708-cb06-4f11-ad14-8d3d160fbc1a
* aws_instance.mongo-replica-01 (deposed #1): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-080ef01dc84c09685' does not exist
    status code: 400, request id: 07c96f82-1e32-4944-a1d6-ab0e6306b82e
* aws_instance.mongo-replica-02 (deposed #1): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-085b997daac742c1e' does not exist
    status code: 400, request id: 20ff2b73-39c9-4d49-af41-f4ec542ec782
* aws_instance.mongo-replica-02 (deposed #0): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-00bc7fd15b04a3688' does not exist
    status code: 400, request id: cdea3c4f-9bec-496a-aedd-bcfbf0a706d2

The AWS EC2 instances in question do not exist indeed, but TF errors out when trying to delete them (presumably because they don't exist) and still keeps them in the state.

I tried using the terraform state command in an attempt to remove the deposed resources, but it only allows me to delete the whole resource, not just the deposed instance of it...

Am I expected to, simply, manually go in the terraform state file and delete the deposed sections??

Answer 1

From the GitHub Issue requesting for Documentation about the "deposed" state

"Deposed" is the state a resource goes into when Terraform is dealing with create_before_destroy... since there can only be one "primary" instance of a given resource, Terraform will first depose the existing one (so it's still tracked in state but no longer used for interpolations), then create the replacement instance before finally deleting the deposed instance.

Errors during that process can cause the deposed instance to stick in the state, since Terraform doesn't want to lose track of them before it can delete them. However, this behavior was a little inconsistent in earlier Terraform versions, and also Terraform handled deposed instances silently rather than including them in the plan output.

So all of this is to say: these deposed instances are leftovers from a failure during the replacement of these resources with create_before_destroy. If you look in the Terraform state file (either terraform.tfstate or .terraform/terraform.tfstate depending on whether you have remote state enabled) you should find the records of these by searching for the word deposed; you can use the data Terraform has stored for these to decide if they are safe to delete before letting Terraform proceed here.

So, you could infer that the "deposed" state of those resources is merely indicative of an underlying issue that's caused errors during the destroy phase. As mentioned in the quote, the "deposed" resources should be "resolved", for lack of a better term, the next time you run terraform apply. In the meantime, check the tfstate files for any/all specific information that terraform has stored to determine if it's safe to proceed and/or to, potentially, diagnose where the problem might be.

Additionally, you can try the following to refresh the local state

terraform refresh      Update local state file against real resources

Also,

terraform debug              Debug output management
terraform state              Advanced state management