This note is a short addition to the last post –CI/CD of Terraform workspace with YAML-based Azure Pipelines. At the end of that article, I mentioned a few alternate routes regarding YAML-based Azure Pipelines, which I will be covering here.
So that we are on the same page, I broke down the build pipeline into four stages: Validate, Dev, Test, and Prod. Under
stage: validate, I added a job that included
terraform init and
terraform validate for all workspaces. Then under
stage: Dev, I added three jobs – plan, approve, and apply. Finally, I followed the same approach for
stage: Test and
After following the build workflow in the
azure-pipelines.yaml file, you must have made at least two observations.
Observation 1: under both the jobs
job: plan and
job: apply (inside stages -Dev, Test, and Prod), I have included a
terraform init task despite having initialized under the
Observation 2: The
terraform plan task (inside
job: plan) does not create a
tfplan file for the
terraform apply task (inside
job: apply) to consume. That means, if there is a state change in the underlying infrastructure from the time the last
terraform plan was run, review and approved, to the time the
terraform apply was run, the plan applied may not be the same plan that was reviewed.
Let me explain that via an example. Say, at a
terraform plan task step, based on the existing underlying infrastructure state, terraform created a list of resources and their desired states. This list of resources and their desired states is known as a plan. The following job is
job: approve where someone reviews the list of resources and their desired states and based on what is required -resumes or rejects the job. While that is being reviewed, say someone else makes a change to the underlying infrastructure state. This state change is not captured in the review process of the previous step. Then, when
terraform apply is run, it won’t make the same state changes that were captured in the
terraform plan task. But instead will take into consideration the state change that was made after the
terraform plan was run. This renders the review process meaningless. The solution is to create the desired state file (aka
tfplan file) at the terraform plan task. And then use the
tfplan file in the
terraform apply step. This approach ensures that plan file that is review is the same plan file that is applied to the underlying infrastructure. However, given that the time in the review stage is only 15 mins in the use case, the chances of an infrastructure state change are relatively low -but it is possible.
I came up with a few options that may address these two observations, and I have listed below the pros and cons of these approaches.
Option 1: No build artifact is created after stage: validate
This is the current state of the
Pros: (i) the approach is secure since no artifacts are accessible to anyone who may have read permission to the build pipeline.
terraform init is run multiple times on the same code base, and (ii) no plan file is created, and hence the observation 2 becomes a concern.
Option 2: Create a build artifact after
Pros: (i) Run
terraform init only once.
Cons :(i) The build artifact is heavy since it includes the
.terraform folder that contains the provider package, and (ii) the build artifact also contains a state file that exposes the
secret access keys that terraform uses to provision resources. As a result, anyone who has read permission to the build pipeline may access the keys. This is a security flaw.
Option 3: Create a build artifact after
stage: validate and create a
tfplan file in
terraform plan step and use that same
tfpan file in the
terraform apply step.
Pros: (i) Run
terraform init only once, and (ii) address the concern raised in observation 2 above.
Cons: Same as the previous option.
All the options mentioned above will work, and we have to choose the one that makes the most sense to us. There can also be variations to the above options to address the “cons” that I have raised. Like, instead of creating a build pipeline artifact, the artifact may be stored in a shared folder with restricted access. Whichever option is chosen, my advice would be to take security concerns very seriously.
I hope you found this note useful. Would you mind commenting if you believe I have missed or misstated something?