Skip to content

Load Balancers

Load balancers are servers that distribute incoming requests to other servers, aiming to balance the load evenly. Why not just handle everything on one server? Simple—it's much more efficient for a server to forward a request than to handle it itself. For example, you can have one load balancer server with 4 vCPUs and four servers with 4 vCPUs each to handle the requests forwarded by the load balancer.

Load Balancer Options

Managed Load Balancer

Most cloud providers offer Load Balancer as a service, which you can use to distribute traffic between servers within the same data center. However, you can't use these with servers from different data centers or with your own hardware because they rely on a physical private network. Another drawback is that TLS (or HTTPS) is usually terminated at the load balancer, meaning the traffic between the load balancer and your servers is often unencrypted. There are also limitations on what you can do with a managed load balancer, not to mention the costs involved.

Self-hosted Load Balancer

A self-hosted load balancer is more flexible because it can distribute requests across servers in different cloud providers, locations, or even a mix of data centers and on-premise machines. However, for such a setup, you'll need a virtual private network (VPN), which we covered in the previous chapter, "Connect Servers with a Virtual Network."

In this chapter, we'll set up a load balancer and two servers with a simple web server that shows the IP address of the server handling each request.

How to Self-Host a Load Balancer

Prerequisites

  • Three servers with public IP addresses, running Ubuntu 22.04, and SSH access. We'll call them "Load Balancer," "Server A," and "Server B." The load balancer will distribute incoming requests between Server A and Server B, while Server A and B will handle the requests and return their IP addresses.
  • A domain or subdomain with a wildcard A record (e.g., *.lb.turbocloud.dev) pointing to the load balancer's public IPv4 address. In this example, I'll use two domains: vpn.lb.turbocloud.dev to create the VPN across all servers and demolb.turbocloud.dev for the test service. A wildcard A record minimizes DNS entries, but feel free to add individual A records.

Steps

  • Follow the steps in the chapter "Connect Servers with a Virtual Network" to create a VPN connecting all three servers. Use the load balancer as the first server (or lighthouse) in the VPN. You only need Caddy Server on the load balancer.

  • Download and start the simple service we’ve prepared for this task on "Server A." The service logs the server's hostname when a request is received and returns the hostname in the response:

ssh root@ip_of_server_a
wget url_to_test_service
chmod +x test_service
./test_service &

  • Repeat the above steps to start the test service on "Server B."

  • Verify that the services are working by opening http://server_a_ip:4004 and http://server_b_ip:4004 in a browser. You should see different IP addresses.

  • Set up the load balancer. SSH into "Load Balancer":

ssh root@ip_of_load_balancer
  • Add a new handler to the Caddyfile for the domain you’re using for the test service (in my case, demo.lb.turbocloud.dev) and reload the Caddyfile. The handler should include the private IPs of Server A and Server B to ensure requests are routed over the VPN instead of using public IPs. No encryption is needed between the load balancer and the servers.
demo.lb.turbocloud.dev {   
    reverse_proxy 192.168.202.1:4004 192.168.202.2:4004 {
        lb_policy round_robin
    }
}
  • That’s it! Now open the domain for your test service (in my case, demolb.turbocloud.dev) multiple times and see if the IP address changes with each refresh. When your project scales to 100 requests per second, each server will handle roughly half of the requests. If you add more servers, each server will handle a smaller portion of the total requests.

Pretty cool, right? You might be wondering, “How do I deploy apps and services to multiple servers?” It’s simple if you’re not using CI: just follow the steps in the chapters "Deployment from Local Machines" and "Deployment from GitHub, Bitbucket, GitLab" for each server, then update the Caddyfile on the load balancer with the app/service domain and private IP addresses of the servers. We’ll cover CI and GitOps for multi-server and multi-cloud setups in a future update.