How to import manual changes into Terraform remote state

Question

I am new to terraform - I have created remote tfstate in s3, and now there are some manual changes too that are done in my AWS infrastructure. I need to import those manual changes into tfstate.

I used the import command for some resources, but for some resources such as IAM policy etc, there is no such import command.

Also some resources such as DB are changed with new parameters added, and I need to import them as well. When I try to import those changes it says:

Error importing: 1 error(s) occurred:

* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide
  with an existing resource.

Please remove or rename this resource before continuing.

Any help would be appreciated. Thanks.

Answer 1

Before directly answering this question I think some context would help:

Behind the scenes, Terraform maintains a state file that contains a mapping from the resources in your configuration to the objects in the underlying provider API. When you create a new object with Terraform, the id of the object that was created is automatically saved in the state so that future commands can locate the referenced object for read, update, and delete operations.

terraform import, then, is a different way to create an entry in the state file. Rather than creating a new object and recording its id, instead the user provides an id on the command line. Terraform reads the object with that id and adds the result to the state file, after which it is indistinguishable in the state from a resource that Terraform created itself.

So with all of that said, let's address your questions one-by-one.

---

Importing Resources That Don't Support terraform import

Since each resource requires a small amount of validation and data-fetching code to do an import, not all resources are supported for import at this time.

Given what we know about what terraform import does from the above, in theory it's possible to skip Terraform's validation of the provided id and instead manually add the resource to the state. This is an advanced operation and must be done with care to avoid corrupting the state.

First, retrieve the state into a local file that you'll use for your local work:

terraform state pull >manual-import.tfstate

This will create a file manual-import.tfstate that you can open in a text editor. It uses JSON syntax, so though its internal structure is not documented as a stable format we can carefully edit it as long as we remain consistent with the expected structure.

It's simplest to locate an existing resource that is in the same module as where you want to import and duplicate and edit it. Let's assume we have a resources object like this:

"resources": {
    "null_resource.foo": {
        "type": "null_resource",
        "depends_on": [],
        "primary": {
            "id": "5897853859325638329",
            "attributes": {
                "id": "5897853859325638329"
            },
            "meta": {},
            "tainted": false
        },
        "deposed": [],
        "provider": ""
    }
},

Each attribute within this resources object corresponds to a resource in your configuration. The attribute name is the type and name of the resource. In this case, the resource type is null_resource and the attribute name is foo. In your case you might see something like aws_instance.server here.

The id attributes are, for many resources (but not all!), the main thing that needs to be populated. So we can duplicate this structure for a hypothetical IAM policy:

"resources": {
    "null_resource.foo": {
        "type": "null_resource",
        "depends_on": [],
        "primary": {
            "id": "5897853859325638329",
            "attributes": {
                "id": "5897853859325638329"
            },
            "meta": {},
            "tainted": false
        },
        "deposed": [],
        "provider": ""
    },
    "aws_iam_policy.example": {
        "type": "aws_iam_policy",
        "depends_on": [],
        "primary": {
            "id": "?????",
            "attributes": {
                "id": "?????"
            },
            "meta": {},
            "tainted": false
        },
        "deposed": [],
        "provider": ""
    }
},

The challenge at this step is to figure out what sort of id this resource requires. The only sure-fire way to know this is to read the code, which tells me that this resource expects the id to be the full ARN of the policy.

With that knowledge, we replace the two ????? sequences in the above example with the ARN of the policy we want to import.

After making manual changes to the state it's necessary to update the serial number at the top-level of the file. Terraform expects that any new change will have a higher serial number, so we can increment this number.

After completing the updates, we must upload the updated state file back into Terraform:

terraform state push manual-import.tfstate

Finally we can ask Terraform to refresh the state to make sure it worked:

terraform refresh

Again, this is a pretty risky process since the state file is Terraform's record of its relationship with the underlying system and it can be hard to recover if the content of this file is lost. It's often easier to simply replace a resource than to go to all of this effort, unless it's already serving a critical role in your infrastructure and there is no graceful migration strategy available.

Imports Colliding With Existing Resources

The error message given in your question is talking about an import "colliding" with an existing resource:

Error importing: 1 error(s) occurred:

* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide with an existing resource.

Please remove or rename this resource before continuing.

The meaning of this message is that when Terraform tried to write the new resource to the state file it found a resource entry already present for the name aws_security_group.Q8SgProdAdminSshInt. This suggests that either it was already imported or that a new security group was already created by Terraform itself.

You can inspect the attributes of the existing resource in state:

terraform state show aws_security_group.Q8SgProdAdminSshInt

Compare the data returned with the security group you were trying to import. If the ids match then there's nothing left to do, since the resource was already imported.

If the ids don't match then you need to figure out which of the two objects is the one you want to keep. If you'd like to keep the one that Terraform already has, you can manually delete the one you were trying to import.

If you'd like to keep the one you were trying to import instead, you can drop the unwanted one from the Terraform state to make way for the import to succeed:

terraform state rm aws_security_group.Q8SgProdAdminSshInt

Note that this just makes Terraform "forget" the resource; it will still exist in EC2, and will need to be deleted manually via the console, command line tools, or API. Be sure to note down its id before deleting it to ensure that you can find it in order to to clean it up.