This note continues the previous one, where I set up HAProxy on an Amazon EC2 instance to route traffic to backend compute instances using Terraform. In that note, the algorithm was round-robin. HAProxy internally manages this as weighted round-robin, and since nothing was configured for the weights, traffic was split evenly across both instances.
In this note, I will show how HAProxy can route traffic to backend Amazon EC2 instances based on CPU utilization. If an instance is under heavy load, HAProxy automatically reduces the traffic it sends to that instance. This is managed via the agent-check mechanism.
Solution Overview
Under this approach, a lightweight agent runs on each backend EC2 instance. The agent listens on a designated port (8888), and when HAProxy connects, it responds with the available CPU capacity as a percentage. HAProxy uses this percentage to adjust the backend’s traffic weight in real time — an instance reporting 20% available capacity receives proportionally less traffic than one reporting 90%.
To demonstrate this, I increased the backend instance count from 2 to 4. After deploying, I first verified that the HAProxy dashboard showed all four backends with equal weight. Then I ran a stress tool on one instance to push its CPU utilization to 80%. Within seconds, the agent reported low availability, and the HAProxy dashboard reflected the reduced weight on that backend.
Prerequisites
This note builds on the first article in this series where I set up HAProxy with round-robin routing. Please go through that first note before proceeding.
The code for this solution is available in my GitHub repository: kunduso/haproxy-ec2-terraform (branch: cpu-based-routing).
Implementation
This use case consists of two steps. These are:
1. Deploy the lightweight agent on the backend EC2 instances, and
2. Update the HAProxy server to route traffic based on agent-reported weights.
Let me now walk you through the implementation.
Step 1: Deploy the lightweight agent on the backend EC2 instances
The logic to install the lightweight agent is in the user-data script run on the backend EC2 instances. The approach separates CPU measurement from the agent response — a background timer calculates CPU availability every 5 seconds and writes it to a file, while the agent simply reads that file when HAProxy connects. This ensures instant responses regardless of system load.
1.1: Create the CPU monitor script
The CPU monitor runs as a systemd timer every 5 seconds. It reads /proc/stat, compares the current values to the previous reading (stored in a separate file), and calculates how much CPU was idle versus busy in that interval. It then writes the available capacity as a percentage to /var/run/haproxy-agent-weight. For example, if the CPU was 80% busy over the last 5 seconds, the file contains 20%.
1.2: Create the agent responder script
The agent responder script (haproxy-agent.sh) reads the pre-calculated weight from the file (/var/run/haproxy-agent-weight) and responds with up XX%. The up keyword explicitly tells HAProxy the server is healthy, and the percentage sets the traffic weight. Because the script only reads a file (no calculation, no sleep), it responds in microseconds.
1.3: Create the systemd units
I created four systemd unit files to wire everything together. The cpu-monitor.service and cpu-monitor.timer pair runs the CPU monitor script every 5 seconds. The haproxy-agent.socket listens on port 8888, and for each incoming connection from HAProxy, systemd spawns the haproxy-agent@.service which runs the agent responder. The StandardInput=socket and StandardOutput=socket directives route the TCP connection directly through the script’s stdin/stdout.
1.4: Start and validate the services
Finally, the user data script reloads systemd, starts and enables both the CPU monitor timer and the agent socket, and validates that the socket is listening. If the socket fails to start, the script exits with an error to prevent a backend from coming up without reporting capability.
Step 2: Update the HAProxy server to route traffic based on agent-reported weights
The change on the HAProxy side is in the haproxy.cfg configuration generated by the user data script. I added weight 100 agent-check agent-port 8888 agent-inter 10s fall 3 rise 2 to each backend server line. This tells HAProxy to:
– Set an explicit base weight of 100 for each backend
– Connect to port 8888 on each backend every 10 seconds and use the response to adjust traffic weight
– Require 3 consecutive agent-check failures before reducing weight to 0 (fall 3)
– Require 2 consecutive successes to restore weight (rise 2)
The existing HTTP health check (check) remains in place — HAProxy runs both: the HTTP health check determines if the server is alive, and the agent-check determines how much traffic it should receive.
The backend security group restricts port 8888 access to only the HAProxy security group — no other source can reach the agent listener. This ensures that the CPU capacity data is only accessible to HAProxy and not exposed to the public internet.
Deployment
The infrastructure is deployed via GitHub Actions using the workflow defined in .github/workflows/terraform.yml. The workflow runs terraform apply only when changes are merged into the main branch, using OIDC authentication to obtain secure, temporary AWS credentials. This ensures that all infrastructure modifications are reviewed via pull requests before deployment.
This repository also includes a code-scanning pipeline (.github/workflows/code-scan.yml) that uses Checkov to scan Terraform configurations for security best practices before deployment. For detailed implementation of Checkov with GitHub Actions, see: automate-terraform-configuration-scan-with-checkov-and-github-actions.
Validation
Validation 1: Verify equal weight distribution
After deploying the infrastructure, I navigated to the HAProxy stats dashboard. The dashboard is accessible on port 8404, which the security group restricts to only my IP address — ensuring that backend health and weight data is not publicly exposed. All four backends showed equal weight, confirming that the agent-check was reporting full capacity on every instance.
Validation 2: Simulate CPU load and observe weight adjustment
I logged into one of the backend Amazon EC2 instances using Session Manager (for details on setting up Session Manager access, see my earlier note). Once connected, I ran stress-ng to push the CPU to 80%:
stress-ng --cpu 0 --cpu-load 80 --timeout 120s --temp-path /tmp
Note: The --temp-path /tmp flag is required when running via Session Manager, as the default working directory is not writable.
Within seconds, the HAProxy dashboard reflected the reduced weight on that backend while the other three maintained their full weight.
In the screenshot above, the stressed backend’s weight dropped to 20% — reflecting only 20% available CPU capacity. HAProxy reduced traffic to this backend proportionally while the remaining backends continued serving at or near full capacity. Traffic was automatically redistributed to the healthy instances with no manual intervention required.
Once the stress run completed and CPU utilization returned to normal, the agent reported full capacity again. Within the next check interval (10 seconds), HAProxy restored the backend’s weight, and the dashboard showed all four instances green with equal weight.
Conclusion
In this note, I demonstrated how HAProxy’s agent-check mechanism enables CPU-based traffic routing across backend Amazon EC2 instances. Unlike static round-robin, this approach adapts to real-time conditions — backends under heavy load automatically receive less traffic and recover their share once the load subsides.
The key components are:
– A CPU monitor script running as a systemd timer that calculates available capacity every 5 seconds
– An agent responder that reads the pre-calculated weight and returns up XX% instantly via systemd socket activation
– The weight 100 agent-check agent-port 8888 agent-inter 10s fall 3 rise 2 directive in HAProxy’s backend configuration
This pattern is useful in environments where backend workloads are uneven — for example, when some instances run batch jobs alongside request handling, or when instance types vary in CPU capacity.
In the next note in this series, I will extend this setup with Auto Scaling Groups and a Lambda function to dynamically update HAProxy’s backend list as instances scale in and out.
If you have any questions or suggestions, feel free to comment or get in touch.
