Per AWS-Docs, AWS Secrets Manager is a service to manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles. Many AWS services store and use secrets in Secrets Manager. In this note, I demonstrate how to access the AWS Secrets Manager secret value using Python from an Amazon EC2 instance.
For that, I:
1. created a couple of secrets and stored those inside AWS Secrets Manager secrets,
2. created a couple of Amazon EC2 instances, and
3. created Python files inside the Amazon EC2 instances using the user data script to access the secret.
I used Terraform to create all these resources. Then, one by one, I logged into the EC2 instances using Session Manager and accessed the secret value using the Python file. I have the code in my GitHub repository so you can follow along if you are interested: ec2-userdata-terraform (please note the branch name: access-secrets-python).
I broke down this use case into eight steps. The first four demonstrate how to create the resources, and the last four show how to access the secret from the Amazon EC2 instances.
Step 1: Create an Amazon VPC and network components to enable internet access
I assigned a small CIDR block to the Amazon VPC and created only one subnet. Then, I created an Internet Gateway and attached that to the Amazon VPC. Finally, I added a route to 0.0.0.0/0 and updated the subnet’s route table to point to that using the Internet Gateway.
There is no need for a route to the internet to test how to access the AWS Secrets Manager secret from an Amazon EC2 instance. I did that because I installed packages on the Amazon EC2 instance and required Sessions Manager access to the Amazon EC2 instances. I required access to the internet for both of these.
Step 2: Create a couple of AWS Secrets Manager secrets
I created two AWS Secrets Manager secrets to store a secret as a string and JSON. I used the random provider’s random_password resource type to create the secrets.
The below image is the secret stored as a string:
This image is the secret stored as a JSON:
Step 3: Create an IAM role that has permissions to access the AWS Secrets Manager secrets
In this step, I created the IAM policy to access the value of the secrets. I created a tight permission policy by providing the ARN of the resources and specific action (secretsmanager:GetSecretValue).
Step 4: Create Amazon EC2 instances and attach the EC2 instance profile
The next step is to create the two Amazon EC2 instances and attach the IAM instance profile. Below is an image of one Amazon EC2 instance’s code:
I passed the AWS Secrets Manager Secret string for the EC2 instance provisioned via the above code; for the other instance, I passed the AWS Secrets Manager Secret JSON.
If you are new to attaching an IAM role to an Amazon EC2 instance, I have a detailed note: Attach an IAM role to an Amazon EC2 instance using Terraform.
I ran the terraform init and terraform apply steps to provision the resources. Please note that if you use my code, please update the backend.tf to store the state file.
Step 5: Log in to the Amazon EC2 instance using Session Manager
One of the permissions policies I attached to the IAM role was arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore, which enabled the Amazon EC2 instance to be accessible from the AWS Console. You need the Amazon EC2 instance to have access to the internet for this to work (or a VPC endpoint for Systems Manager). I have a detailed note on that at: create an Amazon EC2 instance with Session Manager access using Terraform.
After running the Terraform code, you will see that two Amazon EC2 instances were created in your AWS console. Select any one of the instances and click on the Connect button. You will be presented with the below options in the case of Linux.
Select “Session Manager” and click on Connect. You will then have CLI access to the instance.
Step 6: Navigate to the Python file and inspect that
The file is stored in the /var folder as read_secret.py inside the Amazon EC2 instance. You can find that from the user_data script stored in the repository. After you cd into the /var folder using the command cd /var, you can run the ls command. If the Amazon EC2 user data file ran correctly, you must see the read_secret.py file. If interested, you can cat open the file and inspect it. Here is what one of the rendered files looks like:
Please note that it can take 10-12 mins for the user_data script to run and generate the file for you after the Amazon EC2 instance is in the Running state.
Step 7: Run the Python file
Run the Python file using the command python3 read_secret.py, and you must get an output as shown in the below image. The secret value may be different for you since that is randomly generated.
Log in to the other Amazon EC2 instance and run the Python file. This time, you must get a message as below:
Step 8: Verify the AWS Secrets Manager Secret value
On the AWS Console, navigate to AWS Secrets Manager Secret, and you’ll see the two secrets created via this Terraform configuration. Here is the value of the app-1-secret-json secret:
And here is the value of app-1-secret-string secret:
You can see that the secret values match what the Python script displayed.
And that brings us to the end of this note. I hope you understood the use case that I discussed. Please review the user_data template file in the GitHub repository to understand how the Python files were created. Let me know if you have any questions or suggestions.
My Devops Journal 