YAML based Azure Pipeline approach for CI/CD of Terraform workspace

This note is a short addition to the last post –CI/CD of Terraform workspace with YAML-based Azure Pipelines. At the end of that article, I mentioned a few alternate routes regarding YAML-based Azure Pipelines, which I will be covering here.

So that we are on the same page, I broke down the build pipeline into four stages: Validate, Dev, Test, and Prod. Under stage: validate, I added a job that included terraform init and terraform validate for all workspaces. Then under stage: Dev, I added three jobs – plan, approve, and apply. Finally, I followed the same approach for stage: Test and stage: Prod.

After following the build workflow in the azure-pipelines.yaml file, you must have made at least two observations.
Observation 1: under both the jobs job: plan and job: apply (inside stages -Dev, Test, and Prod), I have included a terraform init task despite having initialized under the stage: validate.

Observation 2: The terraform plan task (inside job: plan) does not create a tfplan file for the terraform apply task (inside job: apply) to consume. That means, if there is a state change in the underlying infrastructure from the time the last terraform plan was run, review and approved, to the time the terraform apply was run, the plan applied may not be the same plan that was reviewed.
Let me explain that via an example. Say, at a terraform plan task step, based on the existing underlying infrastructure state, terraform created a list of resources and their desired states. This list of resources and their desired states is known as a plan. The following job is job: approve where someone reviews the list of resources and their desired states and based on what is required -resumes or rejects the job. While that is being reviewed, say someone else makes a change to the underlying infrastructure state. This state change is not captured in the review process of the previous step. Then, when terraform apply is run, it won’t make the same state changes that were captured in the terraform plan task. But instead will take into consideration the state change that was made after the terraform plan was run. This renders the review process meaningless. The solution is to create the desired state file (aka tfplan file) at the terraform plan task. And then use the tfplan file in the terraform apply step. This approach ensures that plan file that is review is the same plan file that is applied to the underlying infrastructure. However, given that the time in the review stage is only 15 mins in the use case, the chances of an infrastructure state change are relatively low -but it is possible.
I came up with a few options that may address these two observations, and I have listed below the pros and cons of these approaches.

Option 1: No build artifact is created after stage: validate
This is the current state of the azure-pipelines.yaml file.
Pros: (i) the approach is secure since no artifacts are accessible to anyone who may have read permission to the build pipeline.
Cons: (i) terraform init is run multiple times on the same code base, and (ii) no plan file is created, and hence the observation 2 becomes a concern.

Option 2: Create a build artifact after stage: validate
Pros: (i) Run terraform init only once.
Cons :(i) The build artifact is heavy since it includes the .terraform folder that contains the provider package, and (ii) the build artifact also contains a state file that exposes the secret access keys that terraform uses to provision resources. As a result, anyone who has read permission to the build pipeline may access the keys. This is a security flaw.

Option 3: Create a build artifact after stage: validate and create a tfplan file in terraform plan step and use that same tfpan file in the terraform apply step.
Pros: (i) Run terraform init only once, and (ii) address the concern raised in observation 2 above.
Cons: Same as the previous option.

All the options mentioned above will work, and we have to choose the one that makes the most sense to us. There can also be variations to the above options to address the “cons” that I have raised. Like, instead of creating a build pipeline artifact, the artifact may be stored in a shared folder with restricted access. Whichever option is chosen, my advice would be to take security concerns very seriously.

I hope you found this note useful. Would you mind commenting if you believe I have missed or misstated something?