ss and conntrack for Connection Debugging

Introduction¶

netstat is deprecated and slow on busy hosts. ss (from iproute2) reads the same kernel data far faster and with better filtering. For anything stateful — NAT, -m state firewall rules, connection limits — you also need conntrack, which exposes the kernel's connection tracking table.

ss essentials¶

ss -tlnp            # TCP, listening, numeric, with process   (the "what's listening" command)
ss -tunap           # TCP + UDP, all states, numeric, process
ss -s               # summary: totals by socket type and TCP state
ss -ti              # TCP with internal info: rtt, cwnd, retransmits, pacing
ss -tn state established '( dport = :443 or sport = :443 )'
ss -tn state time-wait | wc -l
ss -tlnp 'sport = :80'
ss -o state established '( dport = :6379 )'   # -o shows keepalive timer

Reading the queues¶

State    Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN   129    128    0.0.0.0:80          0.0.0.0:*
ESTAB    0      52140  10.0.0.5:443        203.0.113.9:51820

• On a LISTEN socket: Recv-Q = current backlog of completed connections waiting for accept(); Send-Q = the configured backlog (listen()). If Recv-Q is pegged at Send-Q, the app isn't accepting fast enough and clients get resets or timeouts. Raise net.core.somaxconn and the app's backlog argument.
• On an ESTAB socket: Send-Q = bytes sent but unacknowledged. Persistently large = the peer isn't reading, or packet loss / small receive window. Recv-Q = bytes received the app hasn't read() yet — a slow consumer.

conntrack essentials¶

# Current tracked connections
conntrack -L | head
conntrack -L -p tcp --dport 443 | wc -l

# Live event stream (great for "is my rule even matching?")
conntrack -E

# Counts
conntrack -C                    # current entry count
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

Every packet through a state/ct state firewall rule, every NAT translation, and every masqueraded connection consumes a conntrack entry. The table is capped; when it fills, new connections are dropped and the kernel logs nf_conntrack: table full, dropping packet.

# Raise the limit (busy NAT gateway / reverse proxy)
sysctl -w net.netfilter.nf_conntrack_max=524288
# hashsize should be ~ max/8 ; set via module param or:
echo 65536 > /sys/module/nf_conntrack/parameters/hashsize
# persist in /etc/sysctl.d/ and a modprobe.d conf

Tuning timeouts frees entries faster:

# Default TIME_WAIT-ish tracking is 120s; established idle is 5 days (!)
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400

Common scenarios¶

"Cannot assign requested address" from a client making many outbound connections¶

Ephemeral port exhaustion, not conntrack. Check:

sysctl net.ipv4.ip_local_port_range          # widen it: 1024 65535
ss -tn state time-wait | wc -l                # thousands of TIME_WAIT to one dest?

Fix by connection pooling / keep-alive in the app. net.ipv4.tcp_tw_reuse=1 lets the kernel reuse TIME_WAIT sockets for new outbound connections safely. Do not use the old tcp_tw_recycle — it's gone for good reason.

conntrack table full on a NAT gateway¶

dmesg | grep -i conntrack                     # "table full, dropping packet"
conntrack -L -p udp | awk '{print $NF}' | sort | uniq -c | sort -rn | head

Often one host doing a UDP scan or a broken app opening thousands of short UDP flows. Raise nf_conntrack_max, shorten UDP timeouts, and rate-limit or fix the offender. If the box does pure routing with no NAT/stateful rules, you can exempt traffic from tracking with an iptables -t raw ... -j NOTRACK / nftables ct rule.

Verification and troubleshooting¶

• ss -tlnp shows nothing on a port you expect — the service bound to 127.0.0.1 only, or a different network namespace (container). ss -tlnp in the container's netns, or nsenter.
• App logs "connection reset" under load, ss shows Recv-Q full on LISTEN — accept backlog overflow. somaxconn + app backlog + more worker threads.
• nf_conntrack_count near max but few real connections — huge number of stale UDP or SYN_SENT entries; shorten those timeouts.
• conntrack -L is empty on a firewall that clearly filters — the nf_conntrack module isn't loaded (no stateful rules), or you're looking in the wrong namespace.
• Numbers from ss -s don't match conntrack -C — expected. ss counts sockets on this host; conntrack counts flows through it (including forwarded traffic with no local socket).

Related tools and reading¶

• On-site: Port Checker, Ping test, Traceroute.
• Related posts: Finding what is using a port, Tuning open file limits on Linux.