How to reduce the time it takes to refresh Terraform's state?

Question

Most of the AWS infrastructure of the company I work for is described and managed using Terraform.

We have several different services including containerized back-ends and CDN'ed front-ends.

From Route53 domains and namespaces to ELBs, ECS and CloudFront, there is a lot going on.

One of the issues that is happening right now is that, mostly because of the Route53 DNS, checking, refreshing and validating a terraform state takes a long time.

And this is the problem we're trying to solve:

How to drastically reduce the time it takes for tf state to be refreshed/checked?

Moving it into a separate repository apparently is not a good idea because that would make all the Route53 related variables inaccessible or, possibly, outdated.

Answer 1

I came here because I was researching a similar issue. It seems that TF is terrible at graph-walking, so the more interconnected your stuff is, the worse it performs. I have a ball of yarn with 2300 resources. that takes 49 minutes to plan on a machine with enough memory and processors to run at parallelism 10 without peaking. A third is spent refreshing the state, and that can probably not be reduced since it's bound by the AWS CLI calls. But the third spent before state refresh and the third after seem to be mostly TF faffing about in the graph (based on logs).

I found some discussion that would seem to indicate that the structure of your code might influence the planning time dramatically, specifically the use of for_each (link #1 & #2). Since my code base makes heavy use of this, I found that interesting. YMMV ;)

Answer 2

You should break the state out into component sub-states that have sensible logical distinctions, such as "front-end", "caching" or whatever makes sense for how your company organizes and classifies infrastructure.

In terms of making variables accessible, you can declare other states as data sources and pull from them (assuming that they have valid outputs for the values you are interested in).