Troubleshooting Linux 'Neighbour table overflow': Complete ARP Cache & Garbage Collection Tuning Runbook

When managing high-density Linux infrastructure—such as Kubernetes worker nodes, Docker Swarm clusters, KVM virtualization hypervisors, or multi-IP cPanel hosting servers—systems engineers occasionally encounter catastrophic, silent network dropouts. Inbound SSH sessions refuse to connect, DNS resolution fails intermittently, microservices return connection timeouts, and container-to-container traffic drops into a black hole.

A check of dmesg -T or /var/log/messages reveals a recurring kernel alert:

[Tue Oct 14 09:42:11 2026] net_ratelimit: 428 callbacks suppressed
[Tue Oct 14 09:42:11 2026] kernel: Neighbour table overflow.
[Tue Oct 14 09:42:12 2026] kernel: Neighbour table overflow.
[Tue Oct 14 09:42:13 2026] kernel: Neighbour table overflow.

Unlike memory exhaustion (OOM) or CPU starvation, a Neighbour table overflow occurs directly inside the Linux kernel’s Layer 2 to Layer 3 address translation subsystem. When the neighbor cache reaches its hard ceiling, the kernel categorically refuses to create new ARP (IPv4) or NDP (IPv6) entries, dropping every outgoing and routed packet that lacks an existing hardware address mapping.

This engineering guide dissects the internal mechanics of the Linux neighbor subsystem, diagnoses the four primary architectural root causes, and provides an actionable runbook to inspect, flush, and permanently tune neighbor garbage collection parameters for mission-critical enterprise environments.

---

1. Anatomy of the Linux Kernel Neighbor Subsystem

Before modifying sysctl parameters, administrators must understand how the Linux kernel maps Layer 3 IP addresses to Layer 2 Ethernet MAC addresses.

In the Linux network stack, address resolution is managed by the Neighbor Discovery Subsystem:

• IPv4 (ARP): Address Resolution Protocol, managed by the kernel’s ARP module.
• IPv6 (NDP): Neighbor Discovery Protocol, utilizing ICMPv6 router and neighbor solicitations.

Both protocols populate a shared internal data structure: the neighbor table (internally represented by struct neigh_table).

+-------------------------------------------------------------------+
|                     Linux Kernel Network Stack                    |
|                                                                   |
|   Layer 3: IP Routing Decision (IPv4 / IPv6)                      |
|                           |                                       |
|                           v                                       |
|   Neighbor Table Lookup (neigh_lookup / arp_find)                 |
|                           |                                       |
|         +-----------------+-----------------+                     |
|         | Cache Hit                         | Cache Miss          |
|         v                                   v                     |
|   Fast Path (HW MAC attached)        Trigger ARP / NDP Request    |
|   Packet transmitted via NIC         Create entry in INCOMPLETE   |
|                                             |                     |
|                                             v                     |
|                                      Check gc_thresh3:            |
|                                      If current >= gc_thresh3     |
|                                      --> "Neighbour table overflow"|
|                                      --> DROP PACKET IMMEDIATELY  |
+-------------------------------------------------------------------+

The Neighbor State Machine (NUD)

Every entry in the neighbor table cycles through specific Neighbor Unreachability Detection (NUD) states:

• INCOMPLETE: An ARP/NDP request has been broadcasted, but no reply has been received yet.
• REACHABLE: The remote MAC address is confirmed valid within the base_reachable_time_ms window.
• STALE: The validity timer has expired without recent bidirectional confirmation. The kernel can still transmit packets using this MAC, but it schedules a reachability probe.
• DELAY: A packet was sent to a STALE neighbor; the kernel waits a short interval (delay_first_probe_time) before initiating active probing.
• PROBE: The kernel is actively transmitting unicast ARP requests to verify the neighbor.
• FAILED: The neighbor did not respond to probes; address resolution failed.
• PERMANENT: Static, administrator-pinned ARP entries that never expire.

When the neighbor table overflows, the kernel cannot even allocate the initial INCOMPLETE state record. As a result, the kernel drops the packet at layer 2 before it ever leaves the network interface card (NIC).

---

2. The Three Garbage Collection Thresholds Explained

The Linux kernel relies on an internal Garbage Collector (GC) to prune expired, stale, and failed neighbor records. The behavior of this GC engine is dictated by three critical integer thresholds configured per protocol:

gc_thresh1 (The Minimum Inactive Floor)

• Default value: 128
• Behavior: The garbage collector remains completely dormant. As long as the total number of entries is below gc_thresh1, the kernel will never purge entries, even if they transition into STALE or FAILED states.

gc_thresh2 (The Soft Equilibrium Threshold)

• Default value: 512
• Behavior: The soft threshold. When the total number of neighbor records exceeds gc_thresh2, the kernel activates asynchronous background garbage collection (neigh_periodic_work). The collector purges entries that have remained in the STALE state longer than gc_stale_time (default 60 seconds). If entries can be purged, the table stays roughly at or below this level.

gc_thresh3 (The Hard Wall Ceiling)

• Default value: 1024
• Behavior: The hard ceiling. If the neighbor table hits gc_thresh3, the kernel invokes emergency synchronous garbage collection (neigh_forced_gc). If forced garbage collection fails to immediately reclaim table slots (because active entries are REACHABLE or waiting in INCOMPLETE), the kernel halts further allocations. It logs kernel: Neighbour table overflow., increments the discard counter, and drops outgoing packets immediately.

On modern bare-metal systems and virtualized cluster nodes handling thousands of concurrent network connections across local bridge interfaces, default limits dating back to Linux 2.4 kernel defaults (128 / 512 / 1024) are completely insufficient.

---

3. Four Root Causes of ARP Table Exhaustion

Understanding why your system exceeded gc_thresh3 is essential before applying sysctl overrides. In production hosting, four distinct network topographies cause this condition:

Cause 1: High Container and Virtual Machine Density

On a host running hundreds of Docker containers or Kubernetes pods, every container connected to a bridge interface (such as docker0, cbr0, or Calico/Flannel veth interfaces) generates unique MAC addresses and IP endpoints.

When containers communicate across virtual network boundaries or the host routes packets between containers and external subnets, the host’s neighbor table must track:

• The virtual Ethernet MAC of every local container.
• The default gateways of upstream networks.
• Every external IP that pods communicate with directly across flat Layer 2 networks.

A single host hosting 250 pods communicating with 400 external database, Redis, and API endpoints will exceed gc_thresh3 = 1024 within seconds of a traffic spike.

Cause 2: Oversized Layer 2 Broadcast Domains

Datacenter deployments utilizing flat /21 (2,046 hosts) or /22 (1,022 hosts) subnets expose every server interface to an avalanche of broadcast ARP requests.

Every time any server on the subnet broadcasts an ARP request (Who has 192.168.1.50? Tell 192.168.1.1), every connected node’s kernel inspects the packet and creates or updates a temporary neighbor record for the sender. In dense multi-tenant racks, ambient broadcast traffic alone can fill a server’s neighbor table beyond default limits without the server generating outbound traffic of its own.

Cause 3: Network Port Scans and SYN Flood Attacks

When an external attacker or automated vulnerability scanner targets an entire subnet with rapid TCP SYN scans (such as masscan or zmap), the target host attempts to process incoming connections.

If the server initiates outbound responses or probes adjacent addresses, each non-existent target IP forces the kernel to generate an ARP request and create an INCOMPLETE record. If thousands of requests occur within the ARP timeout window, the neighbor table fills with half-open resolution requests, choking out legitimate traffic.

Cause 4: Multi-Homed cPanel & WHM Servers (The ARP Flux Problem)

Multi-homed servers equipped with multiple physical NICs or virtual alias interfaces (common on cPanel/WHM setups with dozens of dedicated IP addresses) frequently fall victim to ARP Flux.

By default, Linux uses a weak host model:

• An ARP request arriving on eth0 asking for an IP assigned to eth1 will be answered by eth0, announcing eth0’s MAC address for an eth1 IP.
• Upstream switches become confused, constantly updating their MAC-to-port lookup tables.
• Neighbor tables on both the server and adjacent gateways churn continuously, preventing the garbage collector from stabilizing active entries.

---

4. Live Diagnostic Runbook: Measuring ARP & NDP Pressure

When network issues occur, execute these diagnostic commands to determine whether neighbor table exhaustion is the true bottleneck.

Step 4.1: Count Current Neighbor Table Entries

Check the active count of IPv4 neighbor entries:

# Count active IPv4 neighbor entries
ip -4 neigh show | wc -l

# Count active IPv6 neighbor entries
ip -6 neigh show | wc -l

# Inspect entry counts grouped by state
ip -4 neigh show | awk '{print $NF}' | sort | uniq -c

A healthy output on a tuned node displays entries predominantly in REACHABLE or STALE states:

     12 DELAY
     48 INCOMPLETE
   1890 REACHABLE
    412 STALE

If the count of INCOMPLETE entries spikes into the hundreds, an active port scan, network loop, or misconfigured routing table is flooding the host.

Step 4.2: Read Kernel ARP Cache Statistics

The Linux kernel maintains runtime neighbor cache counters in /proc/net/stat/arp_cache (for IPv4) and /proc/net/stat/ndisc_cache (for IPv6).

Run this one-liner to format the raw hexadecimal columns into human-readable metrics:

cat /proc/net/stat/arp_cache | awk '
NR==1 {
    for (i=1; i<=NF; i++) header[i] = $i
}
NR>1 {
    print "CPU " (NR-2) ":"
    for (i=1; i<=NF; i++) {
        val = strtonum("0x" $i)
        if (val > 0) printf "  %-22s : %d\n", header[i], val
    }
}'

Key metric indicators:

Counter	Diagnostic Significance
entries	Current number of neighbor entries stored in the kernel table for this CPU core.
allocs	Cumulative count of allocated neighbor entries since system boot.
destroys	Cumulative count of neighbor entries successfully reaped by garbage collection.
forced_gc_runs	Critical alert. Number of times the kernel was forced to run synchronous emergency GC because the table hit gc_thresh3. If this value is incrementing, your server is actively dropping packets.
unres_discards	Critical alert. Number of packets dropped because address resolution could not be completed before table exhaustion or timeout.

If forced_gc_runs and unres_discards are steadily climbing, you must immediately tune the garbage collection thresholds.

---

5. Emergency Remediation Runbook

If your production server is currently experiencing connection drops due to table overflow, perform these emergency runtime adjustments before committing permanent sysctl configurations.

Emergency Command 1: Flush Stale & Failed Neighbor Entries Safely

Rather than purging the entire ARP table—which forces a massive broadcast storm as the server re-resolves every gateway and peer—flush only STALE and FAILED records:

# Safely purge STALE IPv4 entries
ip -4 neigh flush nud stale

# Purge FAILED entries
ip -4 neigh flush nud failed

# Purge INCOMPLETE entries created by port scans
ip -4 neigh flush nud incomplete

If the server is completely unresponsive to external traffic and requires an immediate hard reset of the neighbor table:

# Complete flush of all neighbor entries (Warning: triggers temporary ARP spike)
ip neigh flush all

Emergency Command 2: Increase Thresholds in Live Memory

Immediately raise the kernel thresholds in /proc to stop packet dropping without requiring a reboot:

# Apply IPv4 live overrides
sysctl -w net.ipv4.neigh.default.gc_thresh1=1024
sysctl -w net.ipv4.neigh.default.gc_thresh2=4096
sysctl -w net.ipv4.neigh.default.gc_thresh3=8192

# Apply IPv6 live overrides
sysctl -w net.ipv6.neigh.default.gc_thresh1=1024
sysctl -w net.ipv6.neigh.default.gc_thresh2=4096
sysctl -w net.ipv6.neigh.default.gc_thresh3=8192

Once applied, confirm that dmesg -T | tail -n 20 stops outputting Neighbour table overflow. messages.

---

6. Permanent Sysctl Production Configuration

To ensure these optimizations survive system updates, kernel upgrades, and server reboots, consolidate neighbor table tuning into a dedicated sysctl configuration file.

Create /etc/sysctl.d/99-arp-tuning.conf:

# ====================================================================
# Nextgen High-Density Linux Kernel Neighbor (ARP & NDP) Tuning
# File: /etc/sysctl.d/99-arp-tuning.conf
# ====================================================================

# --------------------------------------------------------------------
# 1. IPv4 ARP Table Capacity & Garbage Collection
# --------------------------------------------------------------------
# Minimum entries before GC activates (dormant threshold)
net.ipv4.neigh.default.gc_thresh1 = 1024

# Soft limit: GC wakes up when stale entries exceed this value
net.ipv4.neigh.default.gc_thresh2 = 4096

# Hard limit: Maximum entries allowed before kernel drops packets
net.ipv4.neigh.default.gc_thresh3 = 8192

# Interval (in seconds) between garbage collection sweeps
net.ipv4.neigh.default.gc_interval = 30

# Time (in seconds) a neighbor entry remains valid in STALE state
net.ipv4.neigh.default.gc_stale_time = 120


# --------------------------------------------------------------------
# 2. IPv6 NDP Table Capacity & Garbage Collection
# --------------------------------------------------------------------
net.ipv6.neigh.default.gc_thresh1 = 1024
net.ipv6.neigh.default.gc_thresh2 = 4096
net.ipv6.neigh.default.gc_thresh3 = 8192
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 120


# --------------------------------------------------------------------
# 3. ARP Flux Mitigation (Essential for Multi-IP & Multi-Homed Servers)
# --------------------------------------------------------------------
# Reply only if target IP address is locally configured on the incoming interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1

# Always use best local address for ARP requests matching the outgoing interface
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

Loading and Verifying the Configuration

Load the new rules into the running kernel:

sysctl --system

Verify that the kernel accepted the parameters:

sysctl net.ipv4.neigh.default.gc_thresh1 \
       net.ipv4.neigh.default.gc_thresh2 \
       net.ipv4.neigh.default.gc_thresh3 \
       net.ipv4.conf.all.arp_ignore \
       net.ipv4.conf.all.arp_announce

Expected output:

net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 4096
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Understanding arp_ignore and arp_announce

For multi-IP cPanel systems and multi-NIC virtualization hosts:

• arp_ignore = 1: Instructs the Linux kernel to only reply to an ARP request if the requested IP address is explicitly bound to the network interface receiving the ARP frame. This stops secondary interfaces from claiming ownership of primary IPs.
• arp_announce = 2: Forces the kernel to use the most appropriate local IP address configured on the egress interface as the sender IP in ARP requests. This prevents gateways from associating the wrong MAC address with secondary aliases.

---

7. Sizing Guidelines for Scale

Different server roles require different neighbor table capacities based on their network architecture. Use the following baseline matrix:

Server Workload / Node Profile	gc_thresh1	gc_thresh2	gc_thresh3	Expected Memory Footprint
Standard cPanel VPS / Single-Tenant Web	512	2048	4096	~1.5 MB Kernel SLAB
Dense Docker Swarm / KVM Node (50-100 VMs)	1024	4096	8192	~3.0 MB Kernel SLAB
Enterprise Kubernetes Worker (250+ Pods)	2048	8192	16384	~6.2 MB Kernel SLAB
High-Throughput Border Gateway / Core Router	4096	16384	32768	~12.5 MB Kernel SLAB

Note: In 64-bit Linux kernels, each neighbor record consumes approximately 384 bytes of non-swappable kernel slab memory (struct neighbour). Raising gc_thresh3 to 8192 costs less than 4 megabytes of RAM—making neighbor table starvation an easily preventable performance risk.

---

8. Architectural Isolation: Why Bare-Metal Dedicated Infrastructure Prevents ARP Pollution

While sysctl tuning resolves immediate table starvation, software workarounds cannot eliminate the architectural drawbacks of shared, oversubscribed Layer 2 network topologies.

In shared virtual environments and multi-tenant cloud VPS clusters:

• Adjacent tenant virtual machines share common virtual switches (vSwitches) and broadcast domains.
• A rogue or compromised VM running scanning tools, misconfigured container bridges, or malfunctioning network daemons can flood the shared hypervisor bridge with ARP broadcasts.
• Virtual network interfaces (such as virtio-net) must continuously handle software interrupts (ksoftirqd), consuming CPU cycles merely parsing irrelevant ARP packets from neighboring tenants.

To achieve true network determinism, enterprise workloads—including high-density Kubernetes worker clusters, large-scale database clusters, and carrier-grade VoIP/streaming platforms—must transition to isolated physical networking.

Deploying on high-performance Dedicated Servers provides dedicated physical switch ports, enterprise Intel or Broadcom network interface controllers with dedicated ring buffers, and isolated 802.1Q VLANs. For mission-critical applications operating in regional business hubs, Dedicated Servers in Pakistan deliver local BGP routing, sub-millisecond local latency, and complete hardware isolation from neighboring tenant broadcast storms.

---

9. Automated Health Check Script

To ensure your operations team detects neighbor table pressure before packet drops occur, deploy this monitoring check as a cron job or monitoring sensor in /usr/local/bin/check-arp-health.sh:

#!/usr/bin/env bash
# ====================================================================
# Nextgen Linux ARP / NDP Neighbor Health Check
# ====================================================================
set -euo pipefail

# Read current capacity and thresholds
CURRENT_ENTRIES=$(ip -4 neigh show | wc -l)
GC_THRESH2=$(sysctl -n net.ipv4.neigh.default.gc_thresh2)
GC_THRESH3=$(sysctl -n net.ipv4.neigh.default.gc_thresh3)

# Check for forced GC runs in procfs
FORCED_GC=$(awk '/forced_gc_runs/ {getline; print strtonum("0x" $10)}' /proc/net/stat/arp_cache 2>/dev/null || echo 0)

# Calculate threshold percentage
USAGE_PCT=$(( CURRENT_ENTRIES * 100 / GC_THRESH3 ))

echo "ARP Table Health Summary:"
echo "  Active Entries : ${CURRENT_ENTRIES}"
echo "  Soft Threshold : ${GC_THRESH2}"
echo "  Hard Limit     : ${GC_THRESH3}"
echo "  Current Usage  : ${USAGE_PCT}%"

if [ "${CURRENT_ENTRIES}" -ge "${GC_THRESH3}" ]; then
    echo "[CRITICAL] Neighbor table overflow detected! Kernel is actively dropping packets."
    exit 2
elif [ "${CURRENT_ENTRIES}" -ge "${GC_THRESH2}" ]; then
    echo "[WARNING] Neighbor table usage exceeds gc_thresh2 (${USAGE_PCT}%). Review garbage collection."
    exit 1
else
    echo "[OK] Neighbor table operating within normal parameters."
    exit 0
fi

Make the script executable:

chmod +x /usr/local/bin/check-arp-health.sh

---

Summary Runbook Checklist

When troubleshooting kernel: Neighbour table overflow:

1. Verify Live Failure: Check dmesg -T | grep -i overflow and inspect /proc/net/stat/arp_cache for climbing forced_gc_runs and unres_discards.
2. Execute Emergency Prune: Run ip -4 neigh flush nud stale and ip -4 neigh flush nud failed to restore immediate connectivity.
3. Apply Runtime Elevation: Temporarily set sysctl -w net.ipv4.neigh.default.gc_thresh3=8192.
4. Deploy Permanent Config: Write /etc/sysctl.d/99-arp-tuning.conf with properly scaled gc_thresh thresholds, gc_interval, and gc_stale_time.
5. Eliminate ARP Flux: Configure arp_ignore = 1 and arp_announce = 2 on multi-IP hosting nodes.
6. Isolate Network Domains: Segment broadcast domains using isolated VLANs and dedicated switch ports on enterprise bare-metal hardware.