Load Balancing Algorithms: Round Robin to Least Connections
A load balancer spreads traffic across backends. Learn the algorithms, the health check, the session affinity, and the layer.
By Velloreth
The purpose
Spread traffic across several backends.
Remove a failed backend from the pool.
Provide a single address for the clients.
The algorithms
| Algorithm | How | Best for |
|---|---|---|
| Round robin | the next in turn | similar requests |
| Weighted round robin | by a weight | unequal servers |
| Least connections | the fewest active | varied requests |
| Least response time | the fastest | a latency goal |
| IP hash | by the source IP | affinity |
| Random | a random pick | a large pool |
| Consistent hash | by a key | a cache |
Round robin is simple and works when the requests are similar.
Least connections adapts when the requests take different times.
Round robin
Request 1 -> backend A
Request 2 -> backend B
Request 3 -> backend C
Request 4 -> backend A
Simple and predictable.
A weighted variant sends more to a larger server.
A: weight 3
B: weight 1
-> A, A, A, B
Least connections
The request goes to the backend with the fewest active connections.
It adapts to a slow request, because a backend that is busy receives less.
It requires the balancer to track the connection count, which is a small cost.
The health check
health_check:
path: /health
interval: 5s
timeout: 2s
unhealthy_threshold: 3
healthy_threshold: 2
A backend that fails the check is removed.
It is added back when it passes.
Without a health check, traffic continues to a failed backend.
A shallow check tests that the process is up, and a deep check tests a dependency.
A deep check that is too aggressive can remove every backend when a shared dependency fails.
Session affinity
Pinning a client to one backend.
The methods:
- A cookie set by the balancer
- The source IP hash
- A header
It is a workaround for a stateful application.
It limits the scaling, because a backend that holds a popular session is loaded.
The better fix is a shared session store.
Layer 4 versus layer 7
| Layer 4 | Layer 7 | |
|---|---|---|
| Routes by | IP and port | the HTTP request |
| Speed | faster | slightly slower |
| TLS | passthrough | termination |
| Routing | limited | by path, host, header |
| Rewrite | no | yes |
A layer 4 balancer is fast and protocol-agnostic.
A layer 7 balancer reads the request, so it can route by the path and terminate TLS.
The TLS
Termination at the balancer means the balancer holds the certificate and the backend sees plain HTTP.
Passthrough means the balancer forwards the encrypted bytes and the backend holds the certificate.
Re-encryption means the balancer terminates and opens a new connection to the backend.
The choice depends on the trust in the internal network and the need for the client certificate.
The draining
A backend that is being removed must drain.
A connection drain waits for the active requests to finish.
A deploy should drain before the process stops, or a request is dropped.
A long-lived connection such as a WebSocket needs a graceful close.
The failure modes
A backend that is slow but not failed is the hardest case, because a health check passes.
A timeout is the guard, and it must be shorter than the client’s.
A retry on another backend is safe for an idempotent request.
A retry storm can amplify a failure, so the retries must be bounded.
Common mistakes
No health check. Traffic to a failed backend.
A health check that is too shallow. A broken backend stays in.
A health check that is too deep. Every backend is removed.
Session affinity by default. A scaling limit.
No connection drain. A dropped request on a deploy.
A retry without a bound. A retry storm.
A timeout longer than the client’s. A wasted request.
One balancer. A single point of failure.
No metrics. The distribution is unknown.
A layer 7 balancer for a non-HTTP protocol. It does not work.
A quotable summary
A load balancer spreads traffic across several backends, removes a failed one with a health check, and provides a single address for the clients. Round robin is simple and least connections adapts to the load, session affinity pins a user to one backend and limits the scaling, and a layer 4 balancer is faster while a layer 7 balancer can route by the path and terminate TLS. Drain the connections before a deploy, and bound the retries so a failure is not amplified.
Related reading
For the related topics, see reverse-proxy-explained and caching-strategies-guide. For the broader context, nginx-basics and how-cdns-actually-work.