In the first part of this series, I showed how to create an HAProxy server on Amazon EC2 and serve traffic to backend EC2 instances using the round-robin algorithm. In the second part, I switched to CPU-metric-based routing, so backend instances with more available CPU capacity received a larger share of traffic than those under heavier load. In the third part, I expanded the idea and replaced the standalone backend instances with an Auto Scaling group. Scaling policies added and removed instances based on the CPUUtilization metric, and an AWS Lambda function kept the HAProxy configuration current so that traffic was always routed to the healthy backend instances.
In all three parts, however, the front end was a single HAProxy EC2 instance, which made it a single point of failure. If that one instance stopped or was terminated, all traffic to the backends would stop. So, I replaced the single HAProxy instance with two instances spread across two Availability Zones and created an Amazon Route 53 DNS record in front of them using a failover routing policy. Under this policy, Route 53 associates a health check with each record and resolves haproxy.kunduso.com to the primary instance while its health check passes. If the primary’s health check fails, Route 53 automatically returns the secondary instance’s IP address instead, so clients continue reaching a working HAProxy instance.
Solution Overview
To address high availability, I created two HAProxy EC2 instances and fronted them with an Amazon Route 53 record. The record gives clients a single, consistent address (haproxy.kunduso.com) regardless of which instance is serving traffic. I created two A records that share this name using a failover routing policy — one marked PRIMARY and the other SECONDARY — each resolving to a different HAProxy instance’s Elastic IP and backed by its own Route 53 health check. Route 53 resolves the name to the primary instance as long as its health check passes. When the primary’s health check fails, Route 53 marks it unhealthy and fails over to the secondary, so clients continue to reach a working HAProxy instance.
Each HAProxy instance is assigned an Elastic IP so that its public address remains stable across stops and restarts. Without this, a stopped and restarted instance would receive a new public IP, and both the Route 53 record and its health check would point to an address that no longer serves traffic.
I also configured a CloudWatch alarm on each Route 53 health check with an Amazon SNS notification, so that when a HAProxy instance is reported unhealthy, the responsible team is alerted rather than relying on someone noticing the failover.
In the previous use cases, the HAProxy security group allowed ingress on port 80 only from my public IP address. In this part, the Route 53 health checkers also need to reach the instances on port 80, so I widened the rule to include the Route 53 health checker IP ranges alongside my own IP, rather than opening it to the entire internet. The stats port (8404) remains restricted to my IP address. I also updated the AWS Lambda function from the previous article to push the HAProxy configuration to both instances instead of one.
Prerequisites
This use-case builds on the three previous articles in this series (linked in the introduction above), so please go through them before proceeding if you have not already.
This use-case also requires a registered domain with an existing Amazon Route 53 public hosted zone. The Terraform configuration looks up the hosted zone using a data source rather than creating it, so the zone must exist before you run terraform plan. I used kunduso.com, and the Terraform configuration created the haproxy.kunduso.com record within it.
The code for this solution is available in my GitHub repository: kunduso/haproxy-ec2-terraform (branch: high-availability-haproxy).
Implementation
This use case has 5 high-level steps. These are:
1. Create the HAProxy EC2 instances
2. Create the Route 53 health checks and records
3. Create the SNS notification
4. Create the CloudWatch metric alarms
5. Update the security group and Lambda function
The steps follow the dependency chain: the health checks reference the Elastic IPs from Step 1, and the CloudWatch alarms in Step 4 reference both the health checks (Step 2) and the SNS topic (Step 3). Let me now walk you through the implementation starting with the HAProxy EC2 instances.
Step 1: Create the HAProxy EC2 instances
The two HAProxy EC2 instances are public-facing. An instance’s default public IP address is released when the instance is stopped and a new one is assigned on the next start, so I attached an Elastic IP to each instance to give it a stable public address that survives a stop and start. This matters because the Route 53 records and health checks point to these addresses.
1.1 Create the Elastic IP addresses
Using the Terraform block above, I created two Elastic IP addresses. The domain = "vpc" property scopes the Elastic IP to this VPC, the only option currently on AWS.
1.2 Create the EC2 instances
This code block creates two HAProxy EC2 instances. The count = var.haproxy_instances property creates two instances, and subnet_id = module.vpc.public_subnets[count.index].id places each one in a different public subnet using count.index. Since those public subnets are in different Availability Zones, the two instances end up spread across two Availability Zones. The rest of the instance configuration — the AMI, IAM instance profile, user data, IMDSv2 metadata options, and encrypted root volume — is unchanged from the first article, so refer to that note for the detailed explanation.
Step 2: Create the Route 53 health checks and records
The aws_route53_health_check resource determines whether the IP address specified in the ip_address property is reachable on the specified port property, which is 80 here. Route 53 health checkers send a request to the resource_path (which is / here) every 10 seconds, as set by the request_interval property.
If the path is not reachable for 3 consecutive checks (failure_threshold = 3, spanning about 30 seconds), Route 53 marks the health check as unhealthy. I created one health check per instance using count, so each HAProxy instance is monitored independently.
I then created two aws_route53_record resources, a primary and a secondary. The haproxy_primary record is an A record in the Route 53 hosted zone with the name haproxy, which combines with the hosted zone’s kunduso.com domain to form haproxy.kunduso.com. Its records point to the Elastic IP of the primary HAProxy EC2 instance, and the secondary record points to the Elastic IP of the second instance.
Both records share the same name (haproxy.kunduso.com), so three properties work together to turn them into an active-passive failover pair:
failover_routing_policy declares the role of each record. The primary record sets type = "PRIMARY" and the secondary sets type = "SECONDARY". This tells Route 53 to answer queries with the primary record while it is healthy, and to use the secondary only when the primary is unhealthy.
set_identifier gives each record a unique label ("primary" and "secondary"). When two or more records share the same name and type, Route 53 requires a set_identifier to tell them apart. Without it, Terraform cannot create a second A record with the same haproxy.kunduso.com name.
health_check_id links each record to the health check created earlier in this step. This is what makes the failover automatic: Route 53 only treats a record as a valid answer while its associated health check is passing. When the primary’s health check fails, Route 53 stops returning the primary record and returns the secondary instead.
Together, these mean a client that resolves haproxy.kunduso.com always receives the IP of a healthy HAProxy instance, and Route 53 handles the switch automatically. I set ttl = 60 on both records so that resolvers cache the answer for only 60 seconds, keeping failover quick by ensuring clients re-query Route 53 and pick up the secondary’s IP soon after the primary is marked unhealthy.
Step 3: Create the SNS notification
In this step, I first created an AWS KMS key to encrypt the SNS notification messages.
The key policy has three statements.
The first grants the account root the standard administrative permissions, which AWS recommends keeping so the key remains manageable.
The second allows the sns.amazonaws.com service principal to generate data keys and encrypt and decrypt the messages, since SNS stores and delivers the messages on the encrypted topic.
The third allows the cloudwatch.amazonaws.com service principal to call kms:GenerateDataKey and kms:Decrypt.
This third statement is the one that is easy to miss: a CloudWatch alarm (explained in Step 4 below) needs both of these actions to publish a notification to a KMS-encrypted topic. If either is missing, the alarm still transitions to the ALARM state, but the publish to SNS fails silently and no email is delivered.
This is also why I used a customer-managed key rather than the default alias/aws/sns AWS-managed key. The managed key’s policy cannot be edited, so it cannot be configured to grant a CloudWatch alarm these permissions, and the alarm action would fail.
Then I created the SNS topic and associated the KMS key to encrypt the messages in the topic.
I also created the aws_sns_topic_policy resource to allow the cloudwatch.amazonaws.com service principal to Publish messages to the SNS topic.
Finally, I added the aws_sns_topic_subscription resource to specify the subscriber when a topic message arrives. In this case I added my email as the endpoint for this SNS topic. This implies that when a notification message arrives, an email will be sent to me with the payload message. Once this setup is deployed, AWS will send a confirmation email to this address, and I have to click the link in that email to confirm the subscription. Until I confirm it, no notifications are delivered.
You will notice that these resources use the Terraform provider in the us-east-1 region although all the other resources are provisioned in us-east-2. I will address that in a bit.
Step 4: Create the CloudWatch metric alarms
The CloudWatch metric alarm continuously watches the HealthCheckStatus metric in the AWS/Route53 namespace, scoped to a specific health check through the HealthCheckId dimension. This metric is 1 when the health check is healthy and 0 when it is unhealthy. I set comparison_operator = "LessThanThreshold" with threshold = 1, so the alarm enters the ALARM state when the metric drops below 1, that is, when the health check reports the instance as unhealthy. When that happens, the alarm publishes to the SNS topic through its alarm_actions, and the ok_actions publish again when the instance recovers, so I am notified of both the failure and the recovery. I created one alarm per instance using count, matching the one health check per instance from Step 2.
I set treat_missing_data = "missing" on the alarm. Route 53 publishes the health check metric continuously, but right after deployment there is a short window with no data points. With missing, the alarm keeps its current state during a data gap instead of treating the absence as a breach. This avoids a false unhealthy alert during that startup window. The alternative, breaching, would treat missing data as a failure and send an unhealthy notification before the metric even starts flowing.
Let me now explain the us-east-1 provider. Although Amazon Route 53 is a global service, it publishes the HealthCheckStatus metric only in the us-east-1 region, regardless of where the rest of my infrastructure runs. Since a CloudWatch alarm can only read metrics in its own region, the alarm has to be created in us-east-1. And because a CloudWatch alarm can only notify an SNS topic in the same region as the alarm, the SNS topic and its KMS key also had to be in us-east-1. That is why I used the us-east-1 Terraform provider for the alarm, the SNS topic, and the KMS key, while the rest of the stack stays in us-east-2.
Step 5: Update the security group and Lambda function
Previously, the security group on the HAProxy instance allowed traffic only from my IP address. Since the HAProxy instances now act as a public-facing load balancer that Route 53 health-checks, port 80 needs to accept traffic from more than just my IP. Instead of opening it to the entire internet (0.0.0.0/0), I scoped it to my IP address and the Route 53 health-checker ranges. An HAProxy instance reachable as an open proxy on the public internet is a security risk, so limiting the sources keeps the load balancer functional for health checks and my own access without exposing it broadly.
I added a new variable holding the Route 53 health checker CIDR ranges.
Then, using for_each with the toset() function, I created an ingress rule per health-checker range to allow TCP traffic on port 80, plus a separate rule for my own IP.
Instead of one, we now have two HAProxy EC2 instances, so I updated the Lambda’s environment variable with a list of those instance IDs.
I also updated the handler.py file so that it pushes the generated configuration to both HAProxy instances instead of one. It sends the SSM command to the list of instance ids and then waits for the command to complete on each instance. The logic that discovers the backend instances and generates the configuration is unchanged from the third note, which I discussed there in detail.
Deployment
The infrastructure is deployed via GitHub Actions using the workflow defined in .github/workflows/terraform.yml. The workflow runs terraform apply only when changes are merged into the main branch, using OIDC authentication to obtain secure, temporary AWS credentials. The OIDC role that the workflow assumes is stored as a GitHub Actions secret (IAM_ROLE), and the notification email address is stored as another secret (NOTIFICATION_EMAIL) that is passed to Terraform as the notification_email variable, so neither the role nor my email address is hardcoded in the repository. This ensures that all infrastructure modifications are reviewed via pull requests before deployment.
This repository also includes a code-scanning pipeline (.github/workflows/code-scan.yml) that uses Checkov to scan Terraform configurations for security best practices before deployment. For detailed implementation of Checkov with GitHub Actions, see: automate-terraform-configuration-scan-with-checkov-and-github-actions.
Two manual steps remain after the first deployment. First, the Lambda function does not fire on the initial deployment because the Auto Scaling group launches its instances before the EventBridge rule is in place to react to those launch events. As a result, the HAProxy configuration starts without any backends, so I manually invoked the Lambda once so it could discover the InService instances and write the initial HAProxy configuration to both HAProxy instances. The Lambda function name follows the ${var.name}-update-haproxy pattern, and I invoked it with the AWS CLI by simulating a launch event.
A successful invocation returns a status code of 200 and a body similar to {"statusCode": 200, "body": "Updated HAProxy with 4 backends"}, confirming that the configuration was pushed to the HAProxy instances. You could instead add this AWS CLI call to the GitHub Actions pipeline, or use an aws_lambda_invocation Terraform resource. Since it is a one-time activity, I ran the AWS CLI manually.
Second, I confirmed the Amazon SNS email subscription by clicking the link in the confirmation email that AWS sent to my address. Until the subscription is confirmed, no notifications are delivered.
I clicked on “Confirm subscription” and was navigated to the confirmation page.
Validation
This use case validates that http://haproxy.kunduso.com stays accessible and continues routing traffic to the backend servers even when the primary HAProxy EC2 instance is stopped. After deployment, I first verified that both Route 53 health checks were healthy and that http://haproxy.kunduso.com routed traffic to the backend EC2 instances.
To observe the failover as it happened, I ran a small loop that polls the endpoint every 15 seconds and prints, on each line, the timestamp, the IP address the name currently resolves to, and the response from the backend. I started this loop before stopping the primary instance so that it captured the behavior before, during, and after the failover:
A couple of details make this loop useful. Resolving the name with dig on every iteration shows the IP changing from the primary Elastic IP to the secondary one, which is the clearest evidence that Route 53 failed over. The --max-time 3 on curl ensures that a request to a stopped instance does not hang and stall the loop. From the Route 53 console I identified the primary HAProxy EC2 instance.
With the loop running, I stopped the primary HAProxy EC2 instance from the AWS Console.
The output below shows the sequence. Note that DNS resolvers cache the record for the ttl of 60 seconds, so there is a short window after the primary is marked unhealthy where a cached response can still point to the stopped instance. The number of failed requests you see depends on this caching window rather than on Route 53 itself, because DNS failover minimizes disruption rather than eliminating it entirely, as the brief failure window below shows.
Reading the output, the endpoint returned the primary Elastic IP (3.138.229.47) with a healthy response for the first few checks. After I stopped the primary instance, there was a window of about one minute and forty-five seconds (roughly 08:04:43 to 08:06:29) where the name still resolved to the primary Elastic IP from cache and requests failed. This window combines the health-check detection time (3 checks at 10-second intervals) and the DNS cache expiry (the 60 second ttl). Once the cached record expired, the loop resolved to the secondary Elastic IP (18.226.134.219) and resumed receiving healthy responses, now served through the second HAProxy instance.
I also confirmed on the Route 53 console that the primary health check had transitioned to unhealthy while the secondary remained healthy.
Finally, the CloudWatch alarm for the primary health check entered the ALARM state and published to the SNS topic. I received the notification email alerting me that the HAProxy instance was unhealthy.
This is the email notification I received, which clearly shows which alarm triggered and when.
That completes the failover half of the test. With the Route 53 health checks and alarms, traffic shifted from the primary to the secondary HAProxy instance, and a notification was sent to inform about the incident.
The second half was to bring the primary HAProxy server back up and confirm the system recovered.
When I restarted the primary instance, its health check returned to healthy after a couple of minutes, Route 53 resumed returning the primary Elastic IP, and the CloudWatch alarm returned to the OK state and sent a recovery notification. Because the instance kept its Elastic IP across the stop and start, the Route 53 record and health check still pointed to a valid address and needed no change. This is worth noting: when you stop and start an EC2 instance that relies on an auto-assigned public IP, it releases that IP and receives a new one on the next start. The instance would still work, but because its IP is referenced both in the Route 53 record and in the health check, both would need to be updated with the new IP address. Associating an Elastic IP avoids this entirely by giving the instance a stable address that survives the stop and start.
The Route 53 console confirmed both health checks back in the “Healthy” state.
A second SNS email confirmed the alarm had returned to the OK state from ALARM.
And that covers the verification. Once the primary recovered, Route 53 resumed resolving haproxy.kunduso.com to the primary HAProxy EC2 instance, and traffic flowed to it again.
Conclusion
In this note, I demonstrated how to add high availability to the HAProxy layer using two instances across two Availability Zones, Amazon Route 53 failover routing, health checks, CloudWatch alarms, and Amazon SNS notifications. When the primary instance becomes unhealthy, Route 53 automatically shifts traffic to the secondary, and the team is notified so a failed instance does not go unnoticed.
This removes the single point of failure from the previous three notes, where a single HAProxy instance meant all traffic stopped if it went down. The failover happens at the DNS level, so it is automatic but not instantaneous — there is a short window governed by the health check interval and the record’s TTL before clients pick up the new address.
In the next note, I will add TLS termination on HAProxy so that the load balancer can serve HTTPS traffic.
If you have questions or suggestions, feel free to comment or reach out.
