This note continues my previous note on Amazon ElastiCache for Redis. In my earlier note, I demonstrated how to create an Amazon ElastiCache for the Redis cluster using Terraform and automate the process using GitHub Actions. In this note, I explain how to connect to the ElastiCache cluster using Python from an Amazon EC2 instance. I have a link to my GitHub repository with the Terraform and Python code.
Prerequisites: Create all the resources discussed in the previous note.
There are three high-level steps to this use case:
Step 1: Create all the supporting additional infrastructure.
Step 2: Log into an Amazon EC2 instance and write into the ElastiCache cluster using Python.
Step 3: Log into another Amazon EC2 instance and read from the ElastiCache cluster using Python.
I will explain these steps in detail. If you are interested in following along, please check my GitHub repository: amazon-elasticache-redis-tf.
Step 1: Create all the supporting additional infrastructure
I discussed the majority of the infrastructure for this use case in my previous note, such as the Amazon Virtual private cloud, subnets, route tables, security groups, Amazon ElastiCache for Redis, and the SSM parameter store to store the ElastiCache endpoint. However, since I am connecting to the ElastiCache cluster from an Amazon EC2 instance using Python, I need additional AWS cloud resources. These are:
-an internet gateway and a route to the internet,
-an IAM policy and instance profile to access the Amazon EC2 instance and
-the Amazon EC2 instances to access the ElastiCache cluster.
This use case does not require internet access. But, since I am using Python3 to access the Amazon ElastiCache for the Redis cluster, there are specific libraries to install on the Amazon EC2 instance, such as python-pip, python3, redis-py-cluster, and boto3. The Amazon EC2 instances require internet access to install these libraries. That is possible by attaching an internet gateway to the Amazon VPC and adding a route in the route table.
So, if you have an Amazon machine image (AMI) with these libraries installed, you do not require an internet gateway or the route.
Since I connected to the ElastiCache cluster from an Amazon EC2 instance, the instance required specific permissions. In my previous note, I created two IAM policies. One had permission to access the ElastiCache endpoint stored in the SSM parameter store, and the other policy had permission to access the ElastiCache auth-token stored in the AWS Secrets Manager secret. I attached both policies to an IAM role that was then attached to the Amazon EC2 instance. Since I was required to access the Amazon EC2 instance via Session Manager, I also attached an AWS-managed policy to the IAM role.
The last AWS cloud resources are the two Amazon EC2 instances. My idea was to create two Amazon EC2 instances, one to write into the ElastiCache cluster and the other to read from the ElastiCache cluster. Using the user data script, I created a Python file and stored it in the var folder of the two Amazon EC2 instances. Hence, both the Amazon EC2 instances use separate user data scripts. I also passed the ElastiCache endpoint address, port number, and AWS Secrets manager secret name to the Python file. Below is an image of one of the Amazon EC2 instances’ Terraform code.
These are all the additional AWS cloud resources required for this use case. I checked-in my code and merged it with the main branch. That triggered the GitHub Actions pipeline and provisioned all the AWS cloud resources.
After all the cloud resources were provisioned, I waited an additional 5 minutes since the Amazon EC2 instances had the user data script to run and install a few packages and create the Python file.
Step 2: Log into an Amazon EC2 instance and write into the ElastiCache cluster using Python
I selected the Amazon EC2 instance named app-4-server-write and clicked on the Connect button to connect via Session Manager. I could log into the Amazon EC2 instance since I had the correct AWS-managed policy and appropriate security group rule attached to the Amazon EC2 instance. You can read more about that at -provision-an-amazon-ec2-instance-with-session-manager-access-using-terraform.
After logging in, I navigated to the /var folder and ran the ls command. I could see the user data generated write_cache.py file.
I ran python3 write_cache.py, which prompted me to enter a City name.
The last message confirmed that the information was successfully stored in the ElastiCache cluster.
Step 3: Log into another Amazon EC2 instance and read from the ElastiCache cluster using Python
I repeated the same steps on the other Amazon EC2 instances – app-4-server-read and logged in via Session Manager. Then, I navigated to the /var folder and ran the Python file python3 read_cache.py. And the correct value was printed. As part of the execution, the Python script connected to the ElastiCache cluster and read the value the other Python script wrote.
You can iterate through Steps 2 and 3 multiple times with different values.
And that brings us to the end of this note. I recommend you cat open the two Python files to review their contents. You will see that the files do not contain any sensitive values, such as the ElastiCache endpoint or the auth_token. Access to those is managed via the IAM role attached to the Amazon EC2 instance.
I hope you found this note helpful. Let me know if you have any questions or suggestions.
My Devops Journal 