Terraform remote state file as a data source to support a layered IAC approach

A few months back, I came across an interesting concept of deploying infrastructure in a layered fashion, and I wish I had the URL saved to refer to it again. Nevertheless, the gist of the idea was that you could build an application product environment by deploying layer after layer of infrastructure. A layer of infrastructure sits on another layer such that the upper layer depends on the lower layer. For example, the first layer could be – create-network-infrastructure. The second layer could be – create-DNS, which is followed by the create-databases layer and create-application-host layer. And finally, the application, as the last layer. Of course, there can be many smaller layers in between too, but you get the concept.

Previously I worked on a similar use case when I created a vpc-peering between two Amazon VPCs that belonged to separate accounts. One of the pre-requisites was that both the Amazon VPCs must exist. So, following the layer conversation from above, the – create-vpc-peering layer sits on top of the create-network-infrastructure layer.

In the previous Terraform configuration project, to create the peering connection, I must provide the VPC IDs of both the Amazon VPCs to the vpc-peering Terraform configuration project. You can read more about it here. However, that option was not scaleable. If I stood up an application product environment from the ground up, the network-infrastructure IDs (VPC ID, etc.) would be different every time. That would imply that I must update the vpc-peering project with new IDs when I require a peering relationship. That does not align entirely with an important IaC capability: re-usable. The vpc-peering configuration project code wouldn’t be re-usable if I provided the vpc IDs each time they changed. The good news is that there is an alternate way to provision cloud resources without sharing the IDs in the code, making it more re-usable. And that is, by using a Terraform state file as a data source. There are pros and cons to that approach and particular situations I cannot use, and I will discuss them in this note.

Note: Yes, instead of sharing the VPC ID in the code as I had in the previous Terraform configuration project, I could have passed it as a command-line variable, and that way, I would not have to store that in a repository. Although it would have made the configuration re-usable, someone still had to fetch the correct IDs and provide them to the Terraform configuration.

I’m assuming that you, the reader, are aware of the data source resource type. You can read more here. Using terraform_remote_state as a data source is quite similar to that. Using that resource block, I can access only those resources that have been exposed in the remote state via the output block. Say, e.g., I am using an existing terraform configuration as a remote state, and that configuration does not have any outputs. In that case, even if I have access to the remote state file, I won’t have to access any information inside the state file. You can read about remote state data sources here.

Let me now walk you through the use case so that you can follow along.
Situation: Two separate Terraform configuration projects created separate Amazon VPCs, a couple of public and private subnets, and route tables, among other resources. Also, both these projects have a remote backend to store the terraform.tfstate files. There are no overlapping CIDR blocks between the two VPCs.

Task: The requirement was to create a peering connection between these two Amazon VPCs such that resources in the public subnets of these two VPCs can communicate with each other.

Approach: Having worked on a similar project in the past, I had a fair idea of creating a peering connection.
Creating a successful peering connection involves four steps – (i) initiate a peering request from the peering owner Amazon VPC, (ii) accept the peering request from the accepting Amazon VPC, (iii) update the peering owner’s route table with CIDR blocks from the accepting VPC/subnet, and (iv) update the accepting VPC route table with CIDR blocks from the peering owner VPC/subnet.

It is also necessary to mention the user credential that I used in this project. The user in the VPC-peering Terraform project needs access to carry out the above four operations in separate AWS accounts. Moreover, since the user also fetches the information (VPC ID, subnet CIDR block, Route table ID, and Account ID) from two separate Terraform remote state files, it requires access to read those state files. I prefer using a restrictive access policy when working with cloud resources. In that regard, I used the assume-role functionality to manage the resources in the two Amazon VPCS that belonged to two separate accounts. You can read about how to create a trusted-trusting relationship across accounts here. I achieved this by attaching the following policies to the user: (i) policy to read the two terraform state files(s3: GetObject on the state file), and (ii) policy to assume a role in both the accounts to create the peering relationship. I also included the policy file in my Github repository alongside the Terraform configuration.

Also, since I’m using the terraform_remote_state data resource, the existing resources (VPCs, subnets, route-tables) need to be:
1. Configured using Terraform.
2. The state file must be remote and accessible.
3. The state file must expose required outputs, namely, VPC ID, account ID, route table ID, and subnet CIDR block.
Here is an example of the output block from the VPC Terraform project whose outputs are accessed in the current VPC-peering Terraform project.

The above two conditions, (i) user having required permission, and (ii) state of existing infrastructure and the state files, are pre-requisites for this approach to work successfully.

Before I get into the steps to create a VPC-peering, I want to discuss how the existing state files are accessed. The link to the GitHub repository is available at: aws-vpc-peering-using-terraform.
I am using two VPCs to create the peering, and one of them is an owner, and the other is the accepter. Hence, while accessing the state files, I named them accordingly. The below image has details.

Using the data "terraform_remote_state" resource type, I can access the state file of the two existing Amazon VPCs. Note that I have to provide the backend and the supporting configuration to access the state file.
After accessing, the state files, for convenience, I added a locals block which made it easier to access the value of the output of the state files.

Let me now walk through the steps involved in creating a peering relationship.

Step 1: Initiate a peering request from the peering owner VPC
I used resource aws_vpc_peering_connection to initiate a peering connection. This information is available at vpc_peering_connection.

Step 2: Accept the peering request from the accepting VPC
I used the resource aws_vpc_peering_connection_accepter to accept the peering connection. This information is available at vpc_peering_connection_accepter.

Step 3 and 4: Update the two route tables with CIDR blocks from the other VPC/subnet

Finally, I used the aws_route resource type to update the route tables in the two VPCs. This information is available at aws_route. Also, note that I had to add a provider to each resource type since these are created in separate AWS accounts. If the peering is between two VPCs in the same account, the provider configuration is not required if the user has permission to manage both VPCs.

Pros and Cons to this approach:
A significant benefit of this approach is that it makes the configuration completely re-usable every time the infrastructure stack has to be re-created. Simultaneously, I had to be careful about what information was being exposed via the output block and provide only that much information, as is required.

Where can this approach not work:
This approach is based on the assumption that the lower-level layers are created using Terraform. If Terraform is not used, then there is no terraform.tfstate file. And if there is no such file, there is nothing for the data "terraform_remote_state" resource type to read off of. In such cases, using the data resource type would be a good idea.

I hope you enjoyed reading this note and found it helpful. Let me know if there are any clarifications necessary.