In the previous two notes, I showed how to set up HAProxy to route traffic using the round-robin algorithm and distribute load based on a CPU agent running on backend Amazon EC2 instances. In both cases, the HAProxy configuration file contained the backend instances’ private IPs at the time of creation. However, in production, workloads typically use Auto Scaling Groups to scale compute capacity based on demand. Instances are launched and terminated dynamically, which means the set of backend IPs changes over time. If the HAProxy configuration is static, it becomes stale the moment an instance is added or removed, resulting in routing traffic to instances that no longer exist or missing newly launched ones entirely.
In this note, I will show how to keep the HAProxy configuration in sync with the Auto Scaling Group using EventBridge, AWS Lambda, and lifecycle hooks. When a new instance reaches "InService" state, Lambda updates the HAProxy configuration to include it. When an instance is selected for termination, Lambda removes it from the configuration before it shuts down — ensuring that no traffic is routed to a terminating backend.
Solution Overview
In this use case, I converted the backend Amazon EC2 instances (from the second use case) into an Auto Scaling Group (ASG) so that the EC2 instances are launched and terminated based on the average CPUUtilization metric in the AWS/EC2 namespace. There are two process flows: (a) a scale-out event where newer Amazon EC2 instances join the Auto Scaling Group, and (b) a scale-in event where older Amazon EC2 instances are terminated in the Auto Scaling Group.
During Scale out: A CloudWatch metric alarm monitors the average CPUUtilization metric in the AWS/EC2 namespace for this ASG. Once the value exceeds 70% for 120 seconds, a scale-out (add Amazon EC2 instance) event is triggered, and a new instance joins the ASG. After a successful instance launch, an EventBridge rule pattern matches the event, triggering an AWS Lambda function. The Lambda function then (a) invokes the autoscaling client to get a list of "InService" instances and their IPs, (b) generates the haproxy.cfg file, and (c) runs the SSM client on the HAProxy EC2 instance to update the configuration. With the haproxy.cfg file updated, HAProxy now routes traffic to the newly added Amazon EC2 instance, as well as to the existing EC2 instances in the ASG.
During Scale in: A CloudWatch metric alarm measures the average CPUUtilization metric in the AWS/EC2 namespace for this ASG. Once the value falls below 30% for 120 seconds, a scale-in (remove Amazon EC2 instance) event is triggered, and an existing instance from the ASG is chosen to be terminated. The ASG’s lifecycle hook pauses the instance termination for up to 300 seconds, placing the instance in Terminating:Wait state. This event matches an EventBridge rule which triggers a Lambda function with the event payload containing the instance to be terminated. The Lambda function queries the ASG for EC2 instances with a LifecycleState of "InService" and excludes the instance to be terminated. It then generates a new haproxy.cfg without that instance and runs the SSM client on the HAProxy EC2 instance to update the configuration. With the haproxy.cfg updated, HAProxy continues routing traffic to the remaining EC2 instances. Finally, the Lambda function invokes the ASG client to complete the lifecycle action, allowing the ASG to terminate the instance.
Prerequisites
This note builds on the two articles in this series where I set up HAProxy. The first one is based on the round-robin routing and the second one is based on an agent running on the backend EC2 instance. Please go through those notes before proceeding.
The code for this solution is available in my GitHub repository: kunduso/haproxy-ec2-terraform (branch: dynamic-backend-discovery).
Implementation
Note: The HAProxy EC2 instance, VPC, security groups, IAM role, and CloudWatch log group remain the same as in the previous notes. The only change is that the haproxy.cfg starts with an empty backend section — the Lambda function populates it after the ASG launches instances.
This solution is built using the following steps:
1. Create an Auto Scaling Group for the backend Amazon EC2 instances
2. Create Amazon CloudWatch alarms and scaling policies for scale in and scale out
3. Create Amazon CloudWatch EventBridge rules and targets
4. Create an AWS Lambda function to query, generate, and update the haproxy.cfg file
Steps 2 and 3 create the triggers that invoke the Lambda function from Step 4 during scaling events.
Let me now explain these in detail starting with the Auto Scaling Group.
Step 1: Create an Auto Scaling Group for the backend Amazon EC2 instances
In this step, I discuss two AWS resources, the launch template and the Auto Scaling Group. The launch template is the blueprint for the Auto Scaling Group, and, hence, contains properties such as the AMI ID, instance type, instance profile, security group IDs, user data, etc.
The Auto Scaling Group uses the launch template properties to create Amazon EC2 instances, along with the subnet to host them on, the health check type, and the maximum, desired, and minimum instance counts. It also includes an initial_lifecycle_hook property that determines the default behavior when the lifecycle transitions to "autoscaling:EC2_INSTANCE_TERMINATING": wait for 300 seconds.
Step 2: Create Amazon CloudWatch alarms and scaling policies for scale in and scale out
I created two sets of Amazon CloudWatch alarms and scaling policies for both scale-out and scale-in activities. The Auto Scaling Group launch template has detailed monitoring enabled via the monitoring {} property; therefore, the CPUUtilization metric is reported to CloudWatch at 1-minute intervals (instead of the default 5-minute intervals).
Once the average CPU utilization exceeds 70% and remains above that threshold for 120 seconds, the alarm triggers, and the alarm_actions property takes effect.
The alarm triggers the aws_autoscaling_policy resource, which scales out (increases) the Amazon EC2 instance count by 1, provided it does not exceed the maximum size of the Auto Scaling Group.
Similarly, once the average CPU utilization drops below 30% and remains below that threshold for 120 seconds, another alarm triggers, and its alarm_actions property takes effect.
And just like the previously discussed aws_autoscaling_policy resource, this one scales in (decreases) the EC2 instance count by 1. The 120-second cooldown prevents another scaling action from triggering immediately after.
For a detailed understanding please check my note on create-an-amazon-ec2-auto-scaling-group-with-metric-scaling-policies-using-terraform.
Step 3: Create Amazon CloudWatch EventBridge rules and targets
Here too, there are two sets of resources, for both the scale out and scale in events. Let me start with the scale out event.
An Amazon CloudWatch Event rule (also known as Amazon EventBridge) matches incoming events and routes them to targets like AWS Lambda, Amazon SNS, or Amazon SQS. It does so via an event pattern.
As you can see from the image above, the event_pattern contains the source and detail type which is matched. Once the pattern matches, a CloudWatch event target is triggered and as you can check in the image below, it triggers an AWS Lambda function.
Amazon EventBridge does not have the permissions to trigger an AWS Lambda function and hence that has to be explicitly defined as shown in the below image.
The action = "lambda:InvokeFunction" sets the type of permission that the principal "events.amazonaws.com" acquires along with the specific AWS Lambda function and the Amazon CloudWatch EventBridge rule.
I created the same set of resources (rule, target, permission) for the scale-in event, matching the EC2 Instance-terminate Lifecycle Action detail type instead.
Step 4: Create an AWS Lambda function to query, generate, and update the haproxy.cfg file
Both the (scale out and scale in) Amazon CloudWatch EventBridge resources trigger the same Lambda function and pass the event payload that contains information of the Auto Scaling group events. The AWS Lambda function reviews the payload and determines whether it’s a scale out or scale in event and generates the haproxy.cfg file.
The Lambda function is triggered after the scale out event. It queries all the InService instances of the Auto Scaling Group and generates the HAProxy config file with the private IPs of the Amazon EC2 instances in the ASG. Since the Lambda function runs after the scale out event, the newly created Amazon EC2 instance’s IP is also included in the query result. Then it utilizes the SSM Run Command to copy the newly generated haproxy.cfg into the HAProxy server. Once HAProxy reloads with the new configuration, traffic starts routing to the newly created Amazon EC2 instance along with the existing ones.
In the case of a scale in event, the ASG lifecycle hook goes into effect which pauses instance termination by 300 seconds. In parallel, the Amazon CloudWatch EventBridge captures the event and triggers the Lambda function with the event payload. The event payload consists of the instance that will be terminated. The Lambda function queries the InService instances of the Auto Scaling Group and removes the instance listed in the payload from the result when it generates the haproxy.cfg. Then the Lambda function copies the newly generated config file using the SSM Run Command mechanism. Once HAProxy reloads with the new configuration, traffic stops routing to the instance that is selected for termination. Lastly, the Lambda function sends the commands to the Auto Scaling Group to continue with instance termination.
Using this logic, the instance chosen for termination stops receiving customer traffic before it gets terminated. This ensures that no customer processes are impacted while in flight. The 300-second lifecycle timeout should be set based on how long typical client sessions last — long enough for existing connections to complete before the instance is terminated.
I’ve explained how to create an AWS Lambda function in a separate note which you can read at automating-AWS-Lambda-deployment.
Deployment
The infrastructure is deployed via GitHub Actions using the workflow defined in .github/workflows/terraform.yml. The workflow runs terraform apply only when changes are merged into the main branch, using OIDC authentication to obtain secure, temporary AWS credentials. This ensures that all infrastructure modifications are reviewed via pull requests before deployment.
This repository also includes a code-scanning pipeline (.github/workflows/code-scan.yml) that uses Checkov to scan Terraform configurations for security best practices before deployment. For detailed implementation of Checkov with GitHub Actions, see: automate-terraform-configuration-scan-with-checkov-and-github-actions.
Validation
After the HAProxy server and backend Amazon EC2 instances were successfully deployed, the HAProxy dashboard displayed the backend Amazon EC2 instances as healthy and the traffic was routing correctly to each instance on every refresh. You can see that from the Sessions Total column in the below image.
To demonstrate a scale out event, I ran stress-ng --cpu 0 --cpu-load 80 --timeout 600s --temp-path /tmp on three of the instances. Using stress-ng I created CPU load on the instances and you can see from the below image that the load was noticed by HAProxy and the backend server weightage was appropriately updated. Notice 20/20 under “Wght” for the three backend servers.
The CPU utilization exceeded 70% and remained elevated for 120 seconds, triggering the Amazon CloudWatch alarm to enter the “In alarm” state.
The subsequent alarm action as set in the resource "aws_cloudwatch_metric_alarm" "cpu_high" {} resource was to scale out and increase the instance count by 1. And we can see that from the ASG activity history from the below screenshot.
After the new instance was added, the HAProxy dashboard immediately started listing that instance in the dashboard. The new instance has the UP status of 23s.
Then, once the stress-ng command completed, the HAProxy dashboard showed all the instances with higher weight.
With no stress-ng running, the CPU utilization dropped and stayed below for 120 seconds which triggered the alarm on the lower side.
The alarm action for the low CPU threshold triggered a scale in, and the same can be seen from the Activity history of ASG.
This also triggered the Lambda function and the selected instance was removed from the HAProxy config file and hence was no longer listed in the dashboard.
The CPU continued to remain below 30%, and one more EC2 instance was terminated after the 120 seconds of cooldown.
Subsequently the Lambda function updated the HAProxy config and dashboard.
This was again followed by one more instance termination since the overall CPU utilization remained below 30%.
And the Lambda function updated the HAProxy config and dashboard as you can see from the dashboard below.
And since I set the min_size property to 2 for resource "aws_autoscaling_group" "backend" {}, the instance count remained at that even though the CPU utilization reported below 30%.
Conclusion
In this note, I demonstrated how to keep HAProxy’s backend configuration in sync with an Auto Scaling Group using EventBridge, AWS Lambda, and lifecycle hooks. As instances scale out, Lambda adds them to HAProxy’s configuration. As instances scale in, Lambda removes them before termination — ensuring no traffic is routed to backends that are about to shut down.
This approach solves the static configuration limitation from the previous two notes. HAProxy no longer needs to know backend IPs at creation time. The configuration updates itself as the infrastructure scales, making it suitable for production workloads with variable demand.
In the next note, I will add high availability to the HAProxy layer itself using Route 53 health checks and failover routing.
If you have any questions or suggestions, feel free to comment or get in touch.
