Terraform Remote State -Part 1: Using AWS

When I started learning Terraform, I installed the tool, wrote Terraform configuration files, and ran the commands from my local (laptop). I did not put a lot of thought into understanding a remote state and how it is useful. As I continued learning, and as I started using Azure DevOps to automate infrastructure provisioning using Terraform, I realized the importance of using a remote state. In this post, I explore the idea of a remote-state in Terraform, why it is useful, and how to set that up. Then I will dive into exploring the remote-state with AWS.

PS: I explore the same idea in my next post (part 2) using Azure for the remote state. In part 3, I explore using a remote-state as a data source. These two posts are being worked on right now.

So that brings us to the question –What is a state? And, what is a remote state?
Let me recap a bit so that we all are on the same page. The sequence of commands in terraform (usually) are init -> validate -> plan -> apply. When “terraform apply” is run, terraform provisions resources as is requested in the configuration files and also stores information about the resource that it just provisioned in order to manage them. Say you changed a config value and re-ran the terraform steps (validate -> plan -> apply). How would the Terraform compiler know that it’s a slight config change to an already provisioned resource and not a new resource request? That information (metadata) is stored in a state file, called terraform.tfstate. By default, this file gets created in the same location from where the Terraform configuration files are executed. Now imagine the challenge when working in an automated, collaborative environment where we cannot take security lightly.

Automation: Say I am using Terraform config files in Azure pipelines hosted build machine to manage resources. Each time the terraform configuration files are run, the terraform.tfstate is created in the local workspace, and then once the build finishes, the workspace gets deleted, and the state file is lost. In the next run (since the previously created tfstate file is not available), terraform provisions resources once again. Depending upon the naming convention or logic, it can create unnecessary duplicate resources erroring out.

Collaboration: Say a team of developers is working on provisioning different infrastructure segments using the same repository to store the Terraform configuration files. In this case, if Developer 1 makes changes and runs the config (whether from local or from a build agent like Azure DevOps), the terraform.tfstate is no longer relevant or is getting lost after each automated run. Why? Because when Developer 2 runs the same Terraform commands, there will be a mismatch in resource configuration. And all this because none of these approaches refers to a shared location of the terraform.tfstate file.

Security: When Terraform creates the terraform.tfstate file after each apply run, it stores meta-data of the resources. This data is sensitive and needs to be stored securely with appropriate access control for only authorized users.

HashiCorp has done a better job of explaining this concept in detail, and I’d recommend reading that here –Terraform state purpose.

I hope that gave you a good understanding of what a state file is and how Terraform uses it. I now explore the idea of a remote-state. A remote-state is a state file that is stored remotely (and not the default -locally). Remotely could be a shared drive as well that is accessible from each developer’s workspace. Terraform also supports Amazon S3, Azure Blog Storage, Terraform cloud, and much more remote-state storage. Although storing Terraform state file remotely is a step in the right direction, it opens up another challenge. What will happen to the remote state file when two parallel processes are trying to update the same terraform.tfstate as part of “terraform apply”? And that is where the concept of state locking is necessary to prevent concurrent access to the state file. Here is a better explanation of remote state and state locking from Terraform Docs.

I now delve into provisioning a remote state and lock for Terraform in AWS. There are two use cases I can think of as a developer explores the idea of using a remote state.
Use Case #1: Fresh terraform configuration. This use-case is when the terraform configurations have not been run, and no resources have been provisioned yet.
Use Case #2: Terraform configs have been used to provision resources and the terraform.tfstate file is currently stored locally.

In the case of AWS, Terraform uses an Amazon S3 bucket and an Amazon Dynamo DB table to store the state file and the lock. That’s a pre-requisite to both the use cases. Below are the steps to add a remote state to a terraform configuration.

Step 1: Add the following block to the terraform configuration file. I have seen repositories (on GitHub) where there is a separate file by the name of “backend.tf” where the backend configuration is stored. I believe storing backend configuration in a separate file keeps the configuration accessible and readable, rather than adding that block to an existing Terraform configuration.

where the $(BackendBucketName) is the name of the Amazon S3 bucket that was created (pre-requisite),
the $(PathToTFStateFile) is the path to the terraform.tfstate file stored inside the bucket, including the file name,
the $(BucketRegion) is the location of the bucket, and
the $(BackendLockTableName) is the name of the Amazon DynamoDB table to store the lock.

Step 2: After the backend block is added to our Terraform configuration, we have to inform Terraform to initiate the backend. That is done with the “terraform init” command. Whether you have an existing terraform.tfstate in local (use-case 2) or running “terraform init” for the first time (use-case 1), we have to start with the “terraform init” command.

The IAM user credentials that will be used by the Terraform compiler to run the Terraform configuration need to have access to the S3 bucket and the DynamoDB table to store the state file and lock.
A policy file can be created, such as shown below, and associated with the user. That ensures that the user will have sufficient permission to work with the backend resources -S3 bucket and DynamoDB table. I know it is easy to give administrator rights to the user, but more than the below is not required to manage the backend.

Here is a link to a repository where I have the backend configured –Terraform-AWS-RemoteBackend.

This repository has a backend.tf file with the details of an Amazon S3 backend. The skeleton code is to configure a VPC, so I added that permission to the IAM user too, along with the policy state above. I also stored the backend-role-policy.json file in the repo. The azure-pipelines.yaml file has all the Terraform steps in detail.

Please note that the Amazon DynamoDB table name, the Amazon S3 bucket name, and the path to the key should match in policy file and the backend configuration. That way, the policy file can be restrictive.

Remote backend in Terraform is a crucial concept to understand as you move towards adopting an infrastructure-as-code mindset. I hope you found this post useful, and let me know if you have any questions or comments.