This note continues the previous one, where I set up HAProxy on an Amazon EC2 instance to route traffic to backend compute instances using Terraform. In that note, the algorithm was round-robin. HAProxy internally manages this as weighted round-robin, and since nothing was configured for the weights, traffic was split evenly across both instances.
In this note, I will show how HAProxy can route traffic to backend Amazon EC2 instances based on CPU utilization. If an instance is under heavy load, HAProxy automatically reduces the traffic it sends to that instance. This is managed via the agent-check mechanism.
Solution Overview
Under this approach, a lightweight agent runs on each backend EC2 instance. The agent listens on a designated port (8888), and when HAProxy connects, it responds with the available CPU capacity as a percentage. HAProxy uses this percentage to adjust the backend’s traffic weight in real time — an instance reporting 20% available capacity receives proportionally less traffic than one reporting 90%.
To demonstrate this, I increased the backend instance count from 2 to 4. After deploying, I first verified that the HAProxy dashboard showed all four backends with equal weight. Then I ran a stress tool on one instance to push its CPU to near 100%. Within seconds, the agent reported low availability, and the HAProxy dashboard reflected the reduced weight on that backend.
Prerequisites
This note builds on the first article in this series where I set up HAProxy with round-robin routing. Please go through that first note before proceeding.
The code for this solution is available in my GitHub repository: kunduso/haproxy-ec2-terraform (branch: cpu-based-routing).
Implementation
This use case consists of two steps. These are:
1. Deploy the lightweight agent on the backend EC2 instances, and
2. Update the HAProxy server to route traffic based on agent-reported weights.
Let me now walk you through the implementation.
Step 1: Deploy the lightweight agent on the backend EC2 instances
The logic to install the lightweight agent is in the user-data script run on the backend EC2 instances. The approach has three sections.
1.1: Create the agent script
The script reads /proc/stat twice, one second apart. This file contains cumulative CPU time counters (user, system, idle, etc.) maintained by the Linux kernel. By comparing the two readings, the script calculates how much of that one-second window the CPU spent idle versus busy. It then reports the available capacity as a percentage — for example, if the CPU was 80% busy during that second, the script returns 20% to HAProxy.
1.2: Create a systemd service
The agent script needs to be listening at all times so that when HAProxy connects, it can respond immediately. I created a systemd service that runs socat — a networking utility that listens on port 8888 and executes the agent script for every incoming connection. The fork option allows it to handle multiple connections without blocking, and Restart=always ensures the listener recovers automatically if it ever crashes.
1.3: Start and validate the service
Finally, the user data script reloads systemd to pick up the new unit file, starts the agent service, and enables it to start on boot. A validation check confirms the service is running — if it fails to start, the user data script exits with an error, preventing a backend from coming up without reporting capability.
Step 2: Update the HAProxy server to route traffic based on agent-reported weights
The only change on the HAProxy side is in the haproxy.cfg configuration generated by the user data script. I added agent-check agent-port 8888 agent-inter 5s to each backend server line. This tells HAProxy to connect to port 8888 on each backend every 5 seconds and use the response to adjust that backend’s traffic weight. The existing health check (check) remains in place — HAProxy now runs both: the HTTP health check determines if the server is alive, and the agent-check determines how much traffic it should receive.
The backend security group restricts port 8888 access to only the HAProxy security group — no other source can reach the agent listener. This ensures that the CPU capacity data is only accessible to HAProxy and not exposed to the public internet.
Deployment
The infrastructure is deployed via GitHub Actions using the workflow defined in .github/workflows/terraform.yml. The workflow runs terraform apply only when changes are merged into the main branch, using OIDC authentication to obtain secure, temporary AWS credentials. This ensures that all infrastructure modifications are reviewed via pull requests before deployment.
This repository also includes a code-scanning pipeline (.github/workflows/code-scan.yml) that uses Checkov to scan Terraform configurations for security best practices before deployment. For detailed implementation of Checkov with GitHub Actions, see: automate-terraform-configuration-scan-with-checkov-and-github-actions.
Validation
Validation 1: Verify equal weight distribution
After deploying the infrastructure, I navigated to the HAProxy stats dashboard. The dashboard is accessible on port 8404, which the security group restricts to only my IP address — ensuring that backend health and weight data is not publicly exposed. All four backends showed equal weight, confirming that the agent-check was reporting full capacity on every instance.
Validation 2: Simulate CPU load and observe weight adjustment
I logged into one of the backend Amazon EC2 instances using Session Manager (for details on setting up Session Manager access, see my earlier note). Once connected, I ran stress-ng to push the CPU to near 100%:
stress-ng --cpu 0 --timeout 240s --temp-path /tmp
Note: The --temp-path /tmp flag is required when running via Session Manager, as the default working directory is not writable.
Within seconds, the HAProxy dashboard reflected the reduced weight on that backend while the other three maintained their full weight.
In the screenshot above, the stressed backend’s weight dropped to 0% — effectively removing it from the traffic pool — while the remaining backends continued serving at full capacity. HAProxy redistributed all traffic to the healthy instances automatically, with no manual intervention required.
Once the stress run completed and CPU utilization returned to normal, the agent reported full capacity again. Within the next check interval (5 seconds), HAProxy restored the backend’s weight, and the dashboard showed all four instances green with equal weight.
Conclusion
In this note, I demonstrated how HAProxy’s agent-check mechanism enables CPU-based traffic routing across backend Amazon EC2 instances. Unlike static round-robin, this approach adapts to real-time conditions — backends under heavy load automatically receive less traffic and recover their share once the load subsides.
The key components are:
– A lightweight bash script that reads /proc/stat to calculate available CPU capacity
– A socat-based systemd service that exposes the script on port 8888
– The agent-check agent-port 8888 agent-inter 5s directive in HAProxy’s backend configuration
This pattern is useful in environments where backend workloads are uneven — for example, when some instances run batch jobs alongside request handling, or when instance types vary in CPU capacity.
In the next note in this series, I will extend this setup with Auto Scaling Groups and a Lambda function to dynamically update HAProxy’s backend list as instances scale in and out.
If you have any questions or suggestions, feel free to comment or get in touch.
