VXLAN and Overlay Networks

Introduction¶

An overlay network carries virtual L2 or L3 networks on top of an existing IP ("underlay") network. VXLAN (Virtual eXtensible LAN) is the dominant encapsulation: it lets you have the same subnet on hosts that are several router hops apart, and gives you 16 million segments instead of VLAN's 4094.

The encapsulation¶

A VXLAN packet wraps the original Ethernet frame:

[ outer Eth ][ outer IP (src/dst = VTEP IPs) ][ outer UDP (dport 4789) ]
  [ VXLAN header: 24-bit VNI ]
    [ original inner Ethernet frame (the VM/pod's actual packet) ]

• VNI (VXLAN Network Identifier) — 24-bit segment ID. VNI 100 on host A and host B is the same virtual L2 domain.
• VTEP (VXLAN Tunnel EndPoint) — the entity that encapsulates/decapsulates. On Linux it's a vxlan interface; on hardware it's a switch ASIC. Each VTEP has an IP in the underlay.
• UDP 4789 (IANA) — Linux historically defaulted to 8472; specify the port.

The underlay just sees UDP packets between VTEP IPs and routes them normally. The inner frame is opaque to it.

The control plane — "how does VTEP A know VM X is behind VTEP B?"¶

• Multicast — each VNI maps to a multicast group; BUM traffic (broadcast, unknown-unicast, multicast) is flooded to the group, and VTEPs learn MAC→VTEP mappings from the inner source. Simple, needs multicast in the underlay (often unavailable, e.g. cloud).
• Head-end replication / static — you configure the list of remote VTEPs; BUM is unicast-replicated to each. Fine for small deployments.
• EVPN (BGP) — VTEPs advertise their local MACs/IPs via MP-BGP (EVPN address family). No flooding, no multicast, scales. This is how modern fabrics and NSX/ACI do it.
• A controller — Kubernetes CNIs (Flannel vxlan backend, Calico vxlan mode, Cilium) have the control plane populate the FDB from the cluster's pod/node data; no multicast, no BGP needed.

Linux example¶

# VTEP IP is 10.0.0.1 here, remote VTEP 10.0.0.2
ip link add vxlan100 type vxlan id 100 dev eth0 \
   local 10.0.0.1 remote 10.0.0.2 dstport 4789
ip link set vxlan100 up
ip addr add 192.168.50.1/24 dev vxlan100
# now 192.168.50.0/24 is a "stretched" segment between the two hosts

For many peers, add FDB entries instead of a single remote:

bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.2   # BUM to this VTEP
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.3

The MTU tax¶

VXLAN adds 50 bytes of overhead (54 with a 802.1Q tag): outer Eth 14 + IP 20 + UDP 8 + VXLAN 8. So on a standard 1500-byte underlay, the overlay MTU is 1450.

• Set the overlay interface / pod MTU to 1450 (or lower for IPsec/WireGuard on top).
• Or enable jumbo frames (9000) on the underlay and keep the overlay at 1500 — the clean solution in a datacenter you control.
• Symptom of getting this wrong: small packets fine, large transfers / TLS handshakes hang (PMTUD blackhole) — the classic overlay MTU bug.

Verification and troubleshooting¶

ip -d link show vxlan100          # VNI, VTEP, dstport
bridge fdb show dev vxlan100      # learned/configured MAC->VTEP
tcpdump -i eth0 -n 'udp port 4789'   # see encapsulated traffic on the underlay
tcpdump -i vxlan100 -n            # see the inner (decapsulated) traffic
ping -M do -s 1422 <overlay peer> # test overlay MTU (1422 + 28 icmp/ip = 1450)

• Overlay peers can't reach each other at all — underlay routing/firewall between VTEP IPs blocking UDP 4789 (or 8472 if that's your port), or VNI mismatch, or the VTEP local IP is wrong.
• ARP works, TCP stalls on big packets — MTU. Lower the overlay/pod MTU by 50 (more for encryption), or jumbo the underlay.
• Intermittent connectivity, MAC on the wrong VTEP — a VM/pod moved and the FDB is stale (multicast/learning mode) — flush and relearn, or the EVPN/CNI control plane didn't update. bridge fdb to inspect.
• Multicast VXLAN silent — the underlay doesn't forward multicast (cloud, or IGMP snooping with no querier). Switch to unicast head-end replication or EVPN.
• Everything slow — encapsulation on the CPU (no NIC offload). Check the NIC supports VXLAN TSO/checksum offload (ethtool -k eth0 | grep tx-udp_tnl) and it's enabled.
• Two overlays with the same VNI leaked into each other — VNI is global to a VTEP; use distinct VNIs per tenant/segment.

Related tools and reading¶

• On-site: Subnet Calculator, Ping test.
• Related posts: 802.1Q VLANs explained, Container networking: bridge, host, none.