CI/CD of Terraform workspace with YAML based Azure Pipelines

This note focuses on using Azure DevOps with Terraform workspace to automate the ideas discussed in the previous two notes (i) CI/CD using Terraform and Azure Pipelines -ideation and (ii) Terraform workspace with multiple AWS accounts. If you are new to Azure DevOps, I would suggest you familiarize yourself with either Azure DevOps build and release definition concepts or Azure Pipelines if you are using YAML builds.

In my previous note (Terraform workspace with multiple AWS accounts), I discussed a use case to provision Amazon VPCs in separate AWS accounts. I will be building on the same use case in this note, using YAML-based Azure Pipelines.

The approach I suggested earlier was to have terraform init and terraform validate run in a CI build definition and have the terraform plan and terraform apply run inside a CD release definition. However, while using terraform workspace I must select a workspace and then run a terraform validate against that particular workspace. Furthermore, in the case of multiple workspaces, I must run terraform validate against all the workspaces.

In such a situation, I tweaked the build algorithm to be as below:
Step 1: Provide the build agent with terraform configuration files and a “list of workspaces” to validate the configuration against
Step 2: Run terraform init (this command would be run against the default workspace)
Step 3: For each workspace provided in the list of workspaces (in step 1), check if the workspace exists and if it does not, create the workspace
Step 4: Validate the selected workspace
Step 5: Repeat steps 3 and 4 until all workspaces specified in the “list of workspaces” are validated.
Step 6: Create a build artifact

Then on the CD side, for each Stage (Dev, Test, and Prod), run the following:
Step 1: Download the build artifact
Step 2: Select a particular workspace
Step 3: Run terraform plan against that workspace
Step 4: Wait for manual validation (reject or resume) of the plan file created in Step 3
Step 5: If step 4 was a resume, run terraform apply

I started on the above approach, but instead of using the classic editor, I chose the YAML-based pipelines. Why? Because the YAML-based pipeline can be versioned and committed to the same code repository where the rest of the terraform configuration files reside. Storing the YAML build definition alongside the code allows someone to clone the code repository and have a running pipeline ready in no time because they’d gain access to the build pipeline too. I have a separate note on my views on YAML based Azure pipelines.

The other benefit of using a YAML-based Azure pipeline was merging the CI-CD into a single Azure pipeline. In the classic editor of Azure DevOps, I could manage only the CI steps in a build definition and create an artifact for the release definition to deploy. However, in the case of Azure pipelines, I could use Stages to implement the workflow of CI and CD both. I believe these concepts are best explained via an example.

I have a working code committed to my Github repo Working-with-Terraform-workspace-and-AWS along with an associated azure-pipelines.yaml file.
The process flow in the azure-pipelines.yaml file is a slight tweak of the one described above. There are stages, and inside that there are jobs, and inside, there are steps, and inside, there are tasks.

The first Stage: Validate has a job: init, which has two tasks:
Task #1: run terraform init
Task #2: run terraform validate using a Powershell script.

The Powershell script (ValidateWorkspace.ps1) takes a “comma-separated list of values” as the parameter, which is the list of workspaces that have to be validated. It then iterates through the list and checks if the workspace is available/created; if not, it creates the workspace. Then it runs terraform validate against that workspace. These are the steps that I planned to include in the build algorithm.

The next three Stages: Dev, Test, and Prod, are pretty identical except for the workspace where they run. Each of these stages is divided into three jobs: plan, approve and apply.

The job: plan is broken down into three tasks:
Task #1: run terraform init
Task #2: select the correct workspace
Task #3: run terraform plan

At the end of task# 3, terraform displays the changes that will be applied to the environment. Then, I introduced another job: approve, which is a manual validation for looking at the plan and resume or reject that. If the plan seems correct, then the validation is resumed, else rejected. If rejected, the pipeline exits. If resumed, the next job: apply starts.

The last job: apply has three tasks:
Task #1: run terraform init
Task #2: select the correct workspace
Task #3: run terraform apply

On successful completion of stage: Dev, the next stage: Test starts, and the same set of jobs and tasks are run. However, the workspace is updated to Test  and, similarly, for stage: Prod where the workspace is Prod.

I have the link to the Azure pipelines build log to check the different stages, jobs, and tasks applied using the azure-pipelines.yaml file (expand the Dev, Test, and Prod stages to view details of the jobs run inside them). Here is an image of the pipeline stages.
 
Not to forget, the terraform state files of all these stages (which are workspaces in terraform parlance) are stored separately. For example, here is an image of the workspace: Dev state file.

As you can see, the path to the state file is Amazon S3/skundu-terraform-remote-state-two/env:/Dev/tf/terraform.tfstate
The state file for workspace: Test is stored at Amazon S3/skundu-terraform-remote-state-two/env:/Test/tf/terraform.tfstate
Where are these values set? Check the backend.tf file in the repository.

And that brings us towards the end of this exciting concept of terraform workspace and how to automate that using Powershell and azure-pipelines.yaml

I ideated on the concept of being able to apply the same terraform configuration across separate environments in my first note. And using terraform workspace I was able to achieve that. However, there is a security observation I want to make. The state files of all the environments (AWS Accounts) are stored in the same Amazon S3 bucket of the trusted account. It might be acceptable for specific organizations. However, I believe the state file of these environments should be stored separately, possibly, in the same account where the infrastructure is provisioned. Either way, whatever decision an organization decides on, they should align with the organization’s security and compliance principles.

PS: While working on the azure-pipeline.yaml file, I identified a few alternate routes possible, and I discussed them in a later note  –YAML based Azure Pipeline approach for CI/CD of Terraform workspace.