Nginx as a Load Balancer

Introduction¶

Beyond reverse-proxying to one backend, Nginx load-balances across a pool with its upstream block. Open-source Nginx does this well for HTTP and, with the stream module, for raw TCP/UDP.

The upstream block¶

upstream app {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=15s weight=2;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=15s;
    server 10.0.1.13:8080 backup;                 # only used when others are down
    keepalive 32;                                  # persistent connections to upstreams
}

server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;
        proxy_set_header Connection "";            # required for upstream keepalive
        proxy_next_upstream error timeout http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_connect_timeout 2s;
        proxy_read_timeout 30s;
    }
}

Balancing methods¶

ip_hash breaks behind a CDN/NAT (many clients → one IP → one backend). Prefer stateless apps; if you need stickiness, use hash on a cookie (hash $cookie_sessionid consistent) or offload sessions to Redis.

Health checks¶

• Open-source Nginx: passive only. max_fails failed responses within fail_timeout marks a server down; after fail_timeout it's retried. There's no active probe — a server that's up but returning garbage stays in rotation until it produces max_fails errors.
• Nginx Plus (or the third-party nginx_upstream_check_module): active health_check with an interval, a URI, and expected status/body.

For real active checks on open-source Nginx, put HAProxy in front of the pool, or use the check module, or rely on your orchestrator (Kubernetes readiness) to pull bad backends.

Retries and proxy_next_upstream¶

proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 3;
proxy_next_upstream_timeout 10s;

If a request to one backend fails with a listed condition, Nginx retries the next backend. Be careful retrying non-idempotent requests — by default Nginx does NOT retry non-GET requests after they've sent a body unless non_idempotent is added. Don't add it unless your backend is truly idempotent.

Upstream keepalive¶

Without keepalive, Nginx opens a new TCP (and TLS) connection to the backend for every request — a lot of overhead at scale. The keepalive 32; + proxy_http_version 1.1; + proxy_set_header Connection ""; trio maintains a pool of persistent connections per worker.

TCP/UDP (L4) with the stream module¶

stream {
    upstream db {
        least_conn;
        server 10.0.2.11:5432 max_fails=2 fail_timeout=10s;
        server 10.0.2.12:5432 max_fails=2 fail_timeout=10s;
    }
    server {
        listen 5432;
        proxy_pass db;
        proxy_connect_timeout 1s;
        proxy_timeout 1h;              # idle timeout for long-lived DB connections
    }

    # UDP (e.g. DNS, syslog)
    server {
        listen 53 udp;
        proxy_pass dns_pool;
        proxy_responses 1;
    }
}

Verification and troubleshooting¶

nginx -T | grep -A10 'upstream app'
curl -s -o /dev/null -w '%{http_code} %{remote_ip}\n' http://lb/ -H 'X-Debug: 1'
# add to the location for debugging:
#   add_header X-Upstream $upstream_addr always;
#   add_header X-Upstream-Status $upstream_status always;
tail -f /var/log/nginx/error.log      # "no live upstreams", "upstream timed out"

• 502 Bad Gateway intermittently — a backend is up but slow/erroring; passive health check hasn't marked it down yet, or max_fails is 0 (disables the check — never do that). Lower max_fails, add proxy_next_upstream.
• no live upstreams while connecting to upstream — all backends marked failed (a transient issue tripped max_fails on all of them, or they're genuinely down). Nginx retries after fail_timeout. Consider a backup server.
• All traffic hitting one backend — ip_hash behind a NAT/CDN, or a hash key with low cardinality. Switch to least_conn/round-robin or hash a higher-cardinality key.
• New connection per request to the backend — keepalive missing, or proxy_http_version still 1.0, or Connection header passed through. All three lines are required.
• Non-idempotent request executed twice — proxy_next_upstream retried a POST. Remove non_idempotent, or make the endpoint idempotent, or don't retry POSTs.
• L4: DB connections drop after 10 min — proxy_timeout (default 10m idle); raise it for long-lived pooled connections.
• Weighted distribution not matching weights — least_conn ignores weight proportions somewhat under uneven load; round-robin honours weights strictly.

Related tools and reading¶

• On-site: Port Checker, HTTP Header Analyzer.
• Related posts: HAProxy basics, An Nginx reverse proxy with TLS.