Issue: Terraform startup time with large statefiles

[rwblokzijl]

Hello!

We have a statefile in which we currently manage 14160 resources. This leads to significant slowdown of terraform commands. We do optimise this with high parellelism, which works well enough. However, we do run into a different scaling issue: Terraform startup time.

When running terraform plan, apply or validate we are currently dealing with 3 minutes of no output before it starts doing anything. This does seem to scale with the size of our statefile. When inspecting our (100MB) statefile, over half of its weight is dependencies. And most of those are transitive dependencies.

1. I am not familiar with the terraform implementation at all, but would there be some optimisations possible here to help us with our usecase?
2. I would also love some suggestions about how to reduce our startup times (we already try to keep depencency counts low)
3. We are planning to split up this statefile, but for that we would like to first see what terraform stacks has to offer. Is there an update about its release date?

[bschaatsbergen]

Hi @rwblokzijl,

Thank you for reporting this. Due to how Terraform needs to track dependencies between resources, I would not expect any huge improvements here in the near future, but there may be some optimizations that could be made.

While it may seem idle, Terraform spends some time building and walking the DAG (Directed Acyclic Graph) it constructs to manage dependencies with, and marshaling the state, sometimes more than once, when you apply for instance. I’ll spare you the finer details, however you can see this for yourself by enabling trace logs.

It could also have something to do with evaluation, and if so, this would be significantly improved by some changes in v1.10.

Also, avoid marking objects with an explicit dependency using depends_on unless it’s really necessary, especially within module calls or when referencing modules, as this can create unnecessary dependencies.

While it’s possible to CPU profile Terraform using pprof, the results usually point to things that the user can’t change.

(FlameGraph of a terraform plan against 10.000 random_pet resources).

[CloudTechnology1]

@rwblokzijl Since your state file is 100MB with significant dependency weight, consider breaking it down further. Each module or service could potentially have its own state file, reducing the overall size and complexity of each individual file. Or Regularly clean up unused resources and outputs in your Terraform configurations to keep the state file lean. Utilize terraform refresh periodically to ensure that your state file reflects the current infrastructure accurately without unnecessary bloat. Thank you