ARP and the Neighbour Table, Explained

Introduction¶

ARP is the glue between layer 3 (IP) and layer 2 (Ethernet). It's invisible until a duplicate IP, a slow failover, or a switch security feature makes it visible — and then the ARP/neighbour table is the first place to look.

Concepts¶

To send an IP packet to a host on the same subnet, your machine needs that host's MAC address. ARP is the "who has 10.0.0.5? tell 10.0.0.9" broadcast; the owner replies with its MAC, and the answer is cached in the neighbour table (historically "ARP cache").

For a destination outside the subnet, the host ARPs for the default gateway instead and sends the frame there.

Cache entry states (Linux):

Gratuitous ARP¶

An unsolicited ARP announcement ("10.0.0.5 is at aa:bb:cc:...") that a host sends about itself. Uses:

• On boot / interface up, to detect duplicate IPs.
• During failover — when a VIP moves to a new node, the new node blasts a gratuitous ARP so every switch and host updates its table and traffic follows immediately. Keepalived, Pacemaker, and cloud "floating IP" mechanisms all do this.

Hands-on¶

# Modern: the neighbour table (ARP is the IPv4 part)
ip neigh show
ip neigh show dev eth0

# Watch it change
ip monitor neigh

# Force resolution of one entry
ip neigh flush 10.0.0.5 && ping -c1 10.0.0.5 && ip neigh show 10.0.0.5

# Pin an entry (rare — a device that doesn't answer ARP)
ip neigh add 10.0.0.9 lladdr aa:bb:cc:dd:ee:ff dev eth0 nud permanent

# Flush the whole table
ip neigh flush all

# Send a gratuitous ARP (e.g. after moving a VIP manually)
arping -U -c3 -I eth0 10.0.0.5

Watch ARP on the wire:

sudo tcpdump -ni eth0 arp

Verification and troubleshooting¶

• Two hosts, same IP — ip neigh flaps between two MACs for that IP, or arping -D -I eth0 10.0.0.5 (duplicate-address detection) gets a reply. arp-scan --localnet lists everything on the segment; match MACs to what you expect.
• Failover works but takes 30+ seconds — the new node isn't sending a gratuitous ARP, so switches keep forwarding to the old MAC until their tables age out. Fix the HA config to announce on takeover.
• Can ping the gateway but nothing beyond — gateway ARP is fine; the problem is routing or the gateway itself, not ARP.
• ip neigh shows FAILED for a host that's up — the host isn't answering ARP: a firewall dropping ARP (unusual), a bonded/bridged interface misconfiguration, or arp_ignore/arp_announce sysctls set aggressively (common with LVS/keepalived on the real servers).
• Intermittent connectivity to one host — STALE entry being revalidated under load, or a switch with Dynamic ARP Inspection dropping packets because there's no matching DHCP snooping binding.
• ARP table full (neighbour: arp_cache: neighbor table overflow in dmesg) — a huge flat subnet or an ARP scan. Raise net.ipv4.neigh.default.gc_thresh{1,2,3}.

Related tools and reading¶

• On-site: Ping, Subnet Calculator, IP Lookup.
• Related posts: DHCP DORA explained, ICMP beyond ping.