TCP Congestion Control Basics

Introduction¶

A TCP sender doesn't just blast data at line rate — it maintains a congestion window (cwnd): how many unacknowledged bytes it's willing to have in flight. Congestion control is the algorithm that grows and shrinks cwnd.

The classic phases¶

Slow start¶

Start with a small cwnd (~10 packets today), double it every RTT (increase by one MSS per ACK). Exponential ramp until either a loss occurs or cwnd reaches ssthresh (slow-start threshold).

Consequence: short connections never leave slow start. A 50 KB response over a 100 ms RTT link takes several round trips regardless of bandwidth — which is why connection reuse (keep-alive, HTTP/2) and TLS session resumption matter so much.

Congestion avoidance¶

After ssthresh, grow cwnd linearly (roughly +1 MSS per RTT) — additive increase.

Loss = multiplicative decrease¶

• Triple duplicate ACK (a packet was lost but later ones arrived) → fast retransmit the missing segment, halve cwnd, continue (fast recovery). Mild.
• Timeout (nothing coming back) → cwnd back to 1, slow start again. Severe.

This "additive increase, multiplicative decrease" (AIMD) sawtooth is classic TCP (Reno/NewReno).

CUBIC — the Linux default¶

CUBIC replaces the linear growth with a cubic function of time since the last loss: grow fast when far from the previous ceiling, cautiously near it, then probe past it. Better than Reno on high-bandwidth, high-latency links ("long fat networks"). Still loss-based — it needs packet loss as the signal to back off.

BBR — model-based¶

BBR (Bottleneck Bandwidth and RTT) doesn't wait for loss. It continuously estimates the bottleneck bandwidth and minimum RTT, and paces sending to bandwidth × min_RTT (the bandwidth-delay product) — keeping the pipe full but the queues empty.

• Much better on lossy paths (Wi-Fi, cellular, lossy long-haul) where CUBIC interprets random loss as congestion and needlessly slows.
• Better at avoiding bufferbloat (see below).
• Can be aggressive toward CUBIC flows sharing a link (a known fairness debate; BBRv2/v3 address it).

Bufferbloat¶

Oversized buffers in routers/modems absorb bursts but also let a loss-based sender keep growing cwnd — the buffer fills with a standing queue, adding hundreds of ms of latency to everything while a bulk transfer runs (the video call stutters while someone uploads).

Fixes: AQM (active queue management) — fq_codel / cake qdisc drops or marks packets before the buffer is full, signalling senders to back off early. BBR also helps because it doesn't fill the buffer in the first place.

# Linux: good defaults for an egress-heavy server / router
net.core.default_qdisc = fq          # or fq_codel
net.ipv4.tcp_congestion_control = bbr

Picking on Linux¶

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control            # current default
# per-socket (an app can also set TCP_CONGESTION)

• CDN origin / video / API to mobile clients: try BBR — usually wins on lossy last miles.
• Datacenter-internal, low loss: CUBIC is fine; DCTCP if you have ECN end-to-end.
• Home router: cake qdisc + set your real uplink/downlink rates — biggest latency win available.

Always A/B test with real traffic; BBR is not universally better.

Verification and troubleshooting¶

ss -tie                          # per-socket: cwnd, rtt, retrans, bbr/cubic state, delivery_rate
ip route show                    # initcwnd/initrwnd if customised
tc qdisc show
nstat -az | grep -Ei 'Retrans|TCPLostRetransmit|TCPTimeouts'

• Throughput far below link capacity on a long-RTT link — loss (CUBIC keeps backing off) or a too-small receive window. ss -tie shows retrans and the window; raise tcp_rmem/tcp_wmem max, try BBR.
• Latency spikes during large transfers — bufferbloat. fq_codel/cake qdisc; test at waveform.com/bufferbloat or with flent.
• BBR enabled, no improvement — path isn't lossy, or the bottleneck is elsewhere (CPU, app, receiver). ss -tie delivery_rate vs expected.
• Short requests slow regardless — slow start + RTT. Not a congestion-control problem; reuse connections, use HTTP/⅔, move content closer (CDN).
• tcp_congestion_control won't change to bbr — module not loaded (modprobe tcp_bbr), or you set it without default_qdisc = fq (BBR needs pacing; fq or kernel-internal pacing).

Related tools and reading¶

• On-site: Ping test, Traceroute.
• Related posts: Understanding MTU and path MTU discovery, sysctl network tuning for servers.