Create an Amazon Managed Grafana dashboard using Terraform and Azure Pipelines

Recently I came across a use case where I was required to create Grafana dashboards using the Terraform Grafana provider. Although it did not sound too complex, it soon became once I started automating the process using Azure Pipelines. In this note, I documented the challenge and the solution.

Before diving deeper, an Amazon Managed Grafana workspace is created using the Terraform AWS provider. Then, a dashboard is created in the Amazon Managed Grafana workspace using the Terraform Grafana provider.

So, it is necessary to have the Amazon Managed Grafana workspace provisioned before working with the dashboard. I covered the process of creating one in a separate note, and you can read about that at: create-an-amazon-managed-grafana-workspace-using-terraform.

As a reader, you might question the decision to separate the deployment of the Amazon Managed Grafana workspace and the dashboard into separate Terraform stacks. And you would be right to ask that. Because by managing two different stacks (one for the workspace and one for the dashboard,) we are increasing the effort to maintain the setup. Before I discuss that, let me walk through the steps to create a dashboard in an Amazon Managed Grafana workspace.

As you follow the below process, you might also question the usage of the PowerShell modules in the pipeline. For example, I used the PowerShell modules to create and delete the Grafana workspace API key. Why did I need the API key? The Terraform Grafana provider requires two values: the URL to the workspace and an API key with an ADMIN role to manage resources (folders, dashboards, etc.) in the Amazon Managed Grafana workspace.
Yes, I know there are alternate ways of generating the API key. However, the process I followed was suitable for my use case.

Note: Terraform must access the AWS Cloud to provision and manage resources. It does so by securely accessing the access_key and the secret_key stored in Azure DevOps. I have the procedure detailed in this post: manage-secure-variables-with-azure-devops-library-and-azure-pipelines.

If you want the code repository handy as you walk through the steps, here’s the link to my GitHub repo:  aws_managed_grafana_workspace_dashboard. You will have to navigate to the amg_dashboard folder for the code on Amazon Managed Grafana dashboard.

Pre-requisite:
Create an Amazon Identity and Access Management (Amazon IAM) user who can manage the resources in this Terraform stack. Store the access_key and secret_key of the user in Azure DevOps Library. If you are cautious and want to use a tight IAM permission policy, I have that in the repository: aws-iam-policy.json. This policy file provides permissions to manage only the Grafana resources in the code. And here is the path to the IAM permission policy to manage the Terraform state file in an s3 bucket: aws-iam-tf-statefile-policy.json. I attached these two policies to the IAM user whose credentials I stored in Azure DevOps Library.
From a skillset standpoint, as a developer, you must know and understand how to create a dashboard in Amazon Managed Grafana workspace using the UI.

Creating a dashboard in an Amazon Managed Grafana workspace using Terraform and CI-CD pipeline can be broken down into five steps:

Step 1: Install the PowerShell modules and create an AWS Profile
Since I am deploying this Terraform stack via Azure DevOps Pipelines and have experience working with PowerShell, my defacto standard while interacting with the AWS cloud resources is AWS Tools for PowerShell. If you have yet to hear or use this, please refer to this note : install-and-use-aws-tools-for-powershell-on-azure-devops-pipelines-yaml.
I was required to use these three PowerShell modules: AWS.Tools.Installer, AWS.Tools.Common, AWS.Tools.ManagedGrafana.
These are specified in the InstallAWSTools.ps1 file stored in the repository.
Using the PowerShell@2 task, I installed the modules and then created an AWS Profile that had the access_key and secret_key of the user.

Step 2: Create an API key for the workspace and store the value in terraform.tfvars
In this step, I used the New-GRFWorkspaceApiKey PowerShell Cmdlet to generate an API key. How was I able to use this? In the previous step, I installed the AWS.Tools.ManagedGrafana PowerShell module that provided me with a list of PowerShell Cmdlets. If you want to learn more, click here and then click on the AWS module. On the new page, scroll down to view the Package Details. All the Cmdlets supported by the module will be listed there.
After creating the API key, I stored the Key value in the variable $apikey and then, using PowerShell, created a string with the grafana_workspace_auth value and the grafana_workspace_url value. Once done, I copied that into the terraform.tfvars file. You will see that the workingDirectory of this step is $(System.DefaultWorkingDirectory)/amg_dasboard. That meant the terraform.tfvars was created in the same directory where the rest of the Terraform source code was stored.

Step 3: Initialize the repository with terraform init
Before getting into the init command, let me discuss the provider.tf file.

You will see that the grafana provider requires two variables. The url and the auth, and in the previous step, I added those into the terraform.tfvars file. The credential (API Key) of the Amazon Managed Grafana workspace was generated as part of a build pipeline step and was not checked-in/committed into the repository.

The first step while working with Terraform code is to initialize the repository, and I did that using the command, as shown below. Since the Grafana provider credentials were stored in the terraform.tfvars file, the default file for storing Terraform variable values, I did not have to pass the variable file in the terraform init command.

Step 4: Create a data source
One of the first steps while working with an Amazon Managed Grafana workspace is to create a data source. As the name suggests, the data source is where the workspace will pull its data to generate graphs and charts. The role assumed by the Amazon Managed Grafana workspace should have permission to read from the data source. The access is managed in the workspace segment of the code. Refer to Step 2 in this note: create-an-amazon-managed-grafana-workspace-using-terraform. 

Step 5: Create a folder and a dashboard
Creating a folder is an optional step. If I had not created a folder, all the dashboards would have been stored in the root folder of the Grafana workspace. Managing access becomes easier if dashboards are stored in different folders.

I created a dashboard using the grafana_dashboard resource type. I copied the Amazon CloudWatch Logs JSON file from grafana-dashboards. That is an excellent source of pre-built dashboards. Alternatively, using the UI, you can create a dashboard in an Amazon Managed Grafana workspace. After finalizing all the panels in the dashboard, you can export the dashboard into a JSON file and add that to your Terraform code.

Step 6: Delete the API key and the AWS PowerShell profile
Once the dashboard was deployed successfully via the Azure DevOps Pipeline, I used the Remove-MGRFWorkspaceApiKey Cmdlet to remove the key created in the earlier step (step 2). Then I also deleted the AWS Profile that I created earlier. There is a condition statement with both these tasks that is set to always(), which implies that even if certain tasks have failed before this task, these two tasks will run. Generally, if steps in a pipeline fail, no further steps are executed unless condition: always() added to that task. That way, I ensure that I do not leave an API key and an AWS Profile un-delete once the pipeline has ended  -successfully or otherwise.

And that brings us to the end of this note on generating a dashboard using Terraform and Azure DevOps.
If you want to dive deeper and access the Azure DevOps build logs, you may find them here: kunduso.aws_managed_grafana_workspace_dashboard.dashboard.

I hope you found this article helpful as you learn more about working with Amazon Managed Grafana, Terraform, and the Grafana Terraform provider. If you have any questions or suggestions, please do not hesitate to reach out.