Few production network anomalies are as baffling or disruptive as the Path MTU (PMTU) Black Hole. The symptoms appear contradictory and defy standard layer-3 ping diagnostics:
# Diagnostic symptom profile
$ ping -c 4 api.internal.enterprise.com
4 packets transmitted, 4 received, 0% packet loss, avg rtt 12.4 ms
$ curl -I https://api.internal.enterprise.com
HTTP/2 200
content-type: application/json
...
$ curl https://api.internal.enterprise.com/v1/export-dataset
* Connected to api.internal.enterprise.com (198.51.100.42) port 443
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [Hangs indefinitely... 120s timeout... Connection timed out]
curl: (28) Failed to connect to api.internal.enterprise.com: Connection timed outA network administrator tests basic connectivity with ping and sees 0% packet loss. They issue curl -I or an HTTP HEAD request, and the server returns headers in milliseconds. But as soon as a real workload passes through—such as an SSL/TLS handshake with a certificate chain exceeding 1,400 bytes, an SSH session executing cat /var/log/syslog, a git clone, or a database query returning multiple rows—the TCP connection freezes completely. No error is returned immediately; packets are retransmitted repeatedly until the kernel hits ETIMEDOUT and terminates the socket.
This failure mode is caused by a PMTU Black Hole. It occurs when packets exceed the Maximum Transmission Unit (MTU) of an intermediate transit link, have the IP Don’t Fragment (DF=1) flag set, and the intermediate router’s fragmentation notification (ICMP Type 3, Code 4) is dropped silently by an upstream firewall, cloud security group, or ISP router.
This runbook delivers an exhaustive, production-grade guide to dissecting the packet-level mechanics of PMTUD failures, sniffing dropped segments with tcpdump, discovering path bottlenecks with stepped payload probing, and implementing permanent mitigations via Linux kernel Packetization-Layer MTU Probing (PLPMTUD), iptables/nftables TCPMSS clamping, and border firewall adjustments.
1. Protocol Mechanics: MTU, MSS, and Path MTU Discovery
To pinpoint where and why large packets vanish, we must inspect the relationship between Layer 2 frames, Layer 3 IP headers, Layer 4 TCP segments, and the Path MTU Discovery (PMTUD) state machine.
+---------------------------------------------------------------------------------------------------+
| Ethernet Frame & TCP Segment Size Calculation |
+---------------------------------------------------------------------------------------------------+
| Standard Ethernet MTU (Maximum Transmission Unit): 1500 Bytes |
| |
| +--------------------+---------------------+--------------------------------------------------+ |
| | IPv4 Header | TCP Header | TCP Payload (Maximum Segment Size - MSS) | |
| | (20 Bytes minimum) | (20 Bytes minimum) | Standard MSS = 1500 - 20 - 20 = 1460 Bytes | |
| +--------------------+---------------------+--------------------------------------------------+ |
| | Optional IP Options| Optional TCP Options| If TCP Timestamps (RFC 7323) active: | |
| | (Up to 40 Bytes) | (TSopt: 12 Bytes) | Effective MSS = 1500 - 20 - 32 = 1448 Bytes | |
| +--------------------+---------------------+--------------------------------------------------+ |
| |
| Encapsulation Overhead on Tunnel Networks: |
| Standard Ethernet : 1500 MTU --> MSS: 1460 Bytes |
| PPPoE (DSL/FTTH) : 1492 MTU --> MSS: 1452 Bytes (8 Bytes PPPoE overhead) |
| VXLAN / Geneve : 1450 MTU --> MSS: 1410 Bytes (50 Bytes outer UDP + VXLAN) |
| WireGuard VPN : 1420 MTU --> MSS: 1380 Bytes (60 Bytes IPv6 or 80 Bytes tunnel overhead) |
| IPsec ESP (IPv4) : 1400 MTU --> MSS: 1360 Bytes (Variable ESP encryption + auth trailer) |
+---------------------------------------------------------------------------------------------------+The IPv4 Fragmentation Mechanics and the DF Flag
In early Internet routing (RFC 791), if a router received an IPv4 packet larger than the outgoing interface MTU, it sliced the packet into fragments, copied the original IP header into each fragment with offset flags, and forwarded them to the destination for reassembly.
However, IP fragmentation introduces massive operational liabilities:
- If a single fragment is dropped in transit, the entire original datagram is lost, forcing a complete TCP retransmission.
- Reassembling fragments consumes CPU and memory on receiving endpoints, opening attack vectors for IP fragmentation denial-of-service attacks (e.g., Teardrop, Ping of Death).
- NAT gateways, stateful firewalls, and Layer-4 load balancers struggle to map fragments lacking Layer-4 port headers to existing connection states.
To eliminate fragmentation, modern operating systems enforce Path MTU Discovery (RFC 1191). The sending Linux kernel sets the Don’t Fragment (DF) bit in the IP header flags field for all outbound TCP packets:
IPv4 Header:
Flags: 0x4000 (Don't Fragment - DF: 1, More Fragments - MF: 0)Classic PMTUD (RFC 1191) Signaling Flow
When a host transmits a packet with DF=1 that exceeds the MTU of any intermediate link, the router handling the link cannot fragment it. By specification, the router must:
- Drop the oversized packet immediately.
- Generate an ICMP Type 3, Code 4 datagram (
Destination Unreachable: Fragmentation Needed and DF set). - Embed the MTU of the next-hop restricting link inside the ICMP payload (per RFC 1191 Section 4).
- Transmit this ICMP message back to the originating host’s source IP address.
+---------------------------------------------------------------------------------------------------+
| RFC 1191 Path MTU Discovery Signaling Sequence |
+---------------------------------------------------------------------------------------------------+
| Client (MTU 1500) Router / Gateway (MTU 1420) Server (MTU 1500) |
| │ │ │ |
| │ 1. TCP SYN (MSS=1460, DF=1) │ │ |
| ├───────────────────────────────────►│─────────────────────────────────────►│ |
| │ │ │ |
| │ 2. TCP SYN-ACK (MSS=1460, DF=1) │ │ |
| │◄───────────────────────────────────│◄─────────────────────────────────────┤ |
| │ │ │ |
| │ 3. TCP ACK (DF=1) │ │ |
| ├───────────────────────────────────►│─────────────────────────────────────►│ |
| │ │ │ |
| │ 4. TLS Client Hello (Size 350B) │ │ |
| ├───────────────────────────────────►│─────────────────────────────────────►│ |
| │ │ │ |
| │ │ 5. TLS Server Hello + Cert (1500B) │ |
| │ │◄─────────────────────────────────────┤ |
| │ │ |
| │ │ [Packet exceeds MTU 1420 & DF=1] |
| │ │ *ROUTER DROPS PACKET* |
| │ │ |
| │ │ 6. ICMP Type 3, Code 4 (Next-Hop MTU = 1420) |
| │ ├─────────────────────────────────────►│ |
| │ │ │ |
| │ │ [Kernel receives ICMP|
| │ │ Updates route cache |
| │ │ Reduces MSS to 1380]|
| │ │ │ |
| │ │ 7. Retransmit Payload (Size 1420B) │ |
| │◄───────────────────────────────────│◄─────────────────────────────────────┤ |
| │ │ │ |
| │ 8. Client receives Certificate │ │ |
| │ Connection proceeds normally │ │ |
+---------------------------------------------------------------------------------------------------+When functioning correctly, the sender’s kernel parses the ICMP packet, extracts the next-hop MTU, updates its internal routing table cache (ip route show cache), calculates a lower MSS, and retransmits the segment. The application layer never notices the change beyond a sub-second latency blip.
2. The PMTU Black Hole Catastrophe
A Path MTU Black Hole occurs when Step 6 in the diagram above fails: the intermediate router drops the oversized packet, but the corresponding ICMP Type 3, Code 4 packet never reaches the sender.
+---------------------------------------------------------------------------------------------------+
| The Anatomy of a PMTU Black Hole |
+---------------------------------------------------------------------------------------------------+
| |
| Sender Intermediate Router Misconfigured Firewall |
| (Linux Host) (Bottleneck MTU) (Drops All ICMP) |
| │ │ │ |
| │ 1. Sends 1500B Data (DF=1) │ │ |
| ├─────────────────────────────────────►│ │ |
| │ │ │ |
| │ │ Packet exceeds MTU │ |
| │ │ *DROPPED* │ |
| │ │ │ |
| │ │ 2. ICMP Type 3, Code 4 │ |
| │ ├──────────────────────────────────►│ |
| │ │ │ *DROPPED SILENTLY*
| │ │ X (Blind ICMP block)
| │ |
| │ 3. TCP Retransmission Timer Expires (RTO) |
| │ 4. Retransmits IDENTICAL 1500B Packet (DF=1) |
| ├─────────────────────────────────────►│ (Dropped again) |
| │ │ |
| │ 5. Exponential Backoff (1s, 2s, 4s, 8s, 16s, 32s...) |
| │ 6. Total Retries Exceeded (tcp_retries2 = 15) |
| │ |
| ▼ |
| [Kernel terminates socket: ETIMEDOUT (Connection timed out)] |
+---------------------------------------------------------------------------------------------------+Why Do Firewalls and ISPs Drop ICMP?
Three pervasive administrative anti-patterns cause PMTU black holes globally:
-
Blind ICMP Filtering (“Security through Obscurity”): Junior security auditors and default firewall scripts frequently apply blanket drops to all incoming ICMP traffic:
# ANTI-PATTERN: Breaks RFC 1191 PMTUD universally iptables -A INPUT -p icmp -j DROPWhile blocking ICMP Echo Requests (
Type 8) mitigates ping sweeps and basic reconnaissance, blocking ICMPType 3(Destination Unreachable) disables essential network control signaling. -
Asymmetric Routing and State Invalidation: In multi-homed or transit BGP topologies, outgoing packets take Route A, while returning ICMP messages take Route B. If Route B has a stateful firewall that inspects incoming ICMP packets without matching state for the forward flow, it rejects the ICMP payload as
INVALID. -
Cloud Security Groups and NAT Gateways: Cloud hypervisors, VPC peering gateways, and edge load balancers often translate external IP headers via SNAT/DNAT. If the NAT gateway fails to rewrite the inner IP header encapsulated inside the ICMP error packet, the receiving Linux host cannot associate the error with its active TCP socket, discarding the notification.
Why Small Packets Succeed While Large Payloads Freeze
This asymmetry confuses engineers because standard troubleshooting tools give false confidence:
- Ping: Default
pingpackets are small (56 data bytes + 8 ICMP header + 20 IP header = 84 bytes), easily traversing links with an MTU of 1400 or lower. - TCP 3-Way Handshake: A SYN packet contains no payload (40 to 60 bytes with options). The initial connection handshake establishes without error.
- HTTP GET / HEAD: The client sends a small HTTP request header (e.g., 200–400 bytes). If the server responds with a short payload (e.g., a
404 Not Foundor301 Redirectof 300 bytes), the entire transaction completes. - The Failure Point: The moment the server attempts to transmit a full MSS payload (e.g., sending a 3,000-byte TLS certificate chain, a database output stream, or an SCP/SFTP data block), the packet size reaches 1,500 bytes. The packet is dropped at the bottleneck, the ICMP notification is discarded, and the connection stalls.
3. Four Core Environments Triggering PMTU Black Holes
In modern multi-cloud and containerized infrastructure, MTUs are rarely uniform across the entire network path. Four environments account for the vast majority of black hole incidents.
Environment 1: Cloud & Hypervisor Overlay Networks
In container platforms and cloud environments, physical networks usually operate at MTU 1500, but overlay networking encapsulates packets inside outer headers:
+---------------------------------------------------------------------------------------------------+
| VXLAN / Overlay Network Encapsulation Stack |
+---------------------------------------------------------------------------------------------------+
| [ Outer Ethernet (14B) ] [ Outer IP (20B) ] [ Outer UDP (8B) ] [ VXLAN (8B) ] |
| Total Encapsulation Overhead = 50 Bytes |
| |
| Physical Interface MTU : 1500 Bytes |
| Maximum Inner Pod MTU : 1500 - 50 = 1450 Bytes |
+---------------------------------------------------------------------------------------------------+- Kubernetes (Calico / Cilium / Flannel): If the host interface MTU is 1500 and the CNI plugin creates
vethinterfaces with an unadjusted MTU of 1500 instead of 1450, pods will transmit 1500-byte packets. The hypervisor tries to encapsulate the packet to 1550 bytes, which exceeds the physical network’s MTU, dropping it unless Jumbo Frames are configured on the underlying switch fabric. - Docker Bridge Networks: By default, Docker assigns
docker0an MTU of 1500. When containers talk to remote APIs across IPsec or WireGuard tunnels with smaller MTUs, outbound TCP connections freeze during payload delivery. - Secondary Cloud vNICs: Secondary interfaces on cloud providers (e.g., Oracle Cloud Infrastructure, AWS EC2, or Azure virtual machines) sometimes enforce an MTU of 9000 within the VPC, but drop to 1500 when egressing to the Internet through an Internet Gateway. Outbound connections to external clients stall if the OS fails to clamp MTU.
Environment 2: VPNs and Encrypted Tunnels
Site-to-site tunnels and road-warrior VPNs are notorious PMTU black holes:
- WireGuard: Adds 32 bytes (IPv4) or 40 bytes (IPv6) of WireGuard framing on top of UDP and IP. Standard setups configure
wg0with an MTU of 1420 bytes (or 1360 bytes on complex mobile carriers). - IPsec ESP: Encapsulating Security Payload adds SPI, sequence numbers, IV, padding, and HMAC authentication. A 1500-byte physical MTU shrinks to an effective MTU of 1360–1400 bytes.
- GRE / SIT Tunnels: Generic Routing Encapsulation adds 24 bytes of overhead.
If a remote office or backup replication server connects via WireGuard, a backup sync over rsync or scp will freeze immediately after authentication unless MSS clamping is explicitly applied to the VPN interface.
Environment 3: Broadband Links in Pakistan (PPPoE / DSL / FTTH)
Broadband providers across Pakistan (including PTCL, Nayatel, StormFiber, and Transworld) terminate residential and small-business optical fiber connections over PPPoE (Point-to-Point Protocol over Ethernet).
- Standard Ethernet frames have an MTU of 1500 bytes.
- The PPPoE header consumes 8 bytes (6 bytes PPPoE + 2 bytes PPP Protocol ID).
- The maximum possible MTU on any standard PPPoE connection is 1492 bytes.
- In turn, the maximum TCP MSS must be 1452 bytes (
1492 - 20 IP - 20 TCP).
When developers, remote sysadmins, or localized payment terminals connect to an overseas VPS or dedicated server:
- The remote server initiates a TLS session with MSS=1460 (assuming standard 1500 MTU).
- The packet hits the local ISP BRAS (Broadband Remote Access Server), where the link MTU is 1492.
- The BRAS drops the 1500-byte packet and sends an ICMP Type 3 Code 4.
- Upstream transit firewalls or the hosting provider’s edge filter drop the ICMP packet.
- The local user’s browser spins forever on “Establishing Secure Connection…”.
Environment 4: Strict Border Firewalls and Security Suites
Many popular hosting security tools ship with default configurations that inadvertently trigger black holes:
- ConfigServer Security & Firewall (CSF): Setups with overly aggressive
ICMP_IN = "0"drop all ICMP packets without distinction. - UFW (Uncomplicated Firewall): Custom iptables chains inserted before
/etc/ufw/before.rulesthat blindly drop ICMP. - Raw iptables scripts: Systems configured with naive rules such as:
This immediately isolates the host from Path MTU updates sent by intermediate transit routers worldwide.iptables -I INPUT -p icmp -j DROP
4. Diagnostic Procedures & Real-Time Packet Analysis
When troubleshooting a suspected PMTU black hole, standard ping and traceroute commands can be misleading. You must systematically probe the path MTU, inspect the kernel route cache, sniff raw network traffic, and trace TLS handshakes.
+---------------------------------------------------------------------------------------------------+
| PMTU Diagnostic Decision Flowchart |
+---------------------------------------------------------------------------------------------------+
| |
| [Symptom: TCP Handshake OK, Large Request/Response Hangs] |
| │ |
| ▼ |
| Step 1: Test Path MTU with Stepped Payloads (ping -M do -s <size>) |
| │ |
| ├── Payload 1472B (1500 MTU) Fails? ──► Potential Bottleneck on Path |
| └── Step down: 1464 (1492) -> 1392 (1420) -> 1372 (1400) |
| Pinpoint exact bottleneck MTU. |
| │ |
| ▼ |
| Step 2: Inspect Kernel Route Cache (ip route get <destination_ip>) |
| │ |
| └── Check if 'mtu' or 'advmss' is cached and whether it updates. |
| │ |
| ▼ |
| Step 3: Capture Network Traffic (tcpdump) |
| │ |
| ├── Check for ICMP Type 3 Code 4 packets: icmp[0] == 3 and icmp[1] == 4 |
| └── Check for repeated TCP Retransmissions with identical sequence numbers. |
| │ |
| ▼ |
| Step 4: Trace Application Latency (curl -Iv --trace-time) |
| │ |
| └── Confirm freeze occurs precisely during TLS Server Hello / Certificate exchange. |
+---------------------------------------------------------------------------------------------------+Procedure 1: Pinpoint the Bottleneck MTU Using ping with the DF Bit Set
Standard ping tools allow setting the IP Don’t Fragment flag manually. On Linux, use the -M do flag combined with -s to specify payload size.
Remember the header overhead calculation: $$\text{Total Packet Size (MTU)} = \text{ICMP Payload Size} + 8\text{ bytes (ICMP header)} + 20\text{ bytes (IP header)}$$ $$\text{ICMP Payload Size} = \text{Target MTU} - 28$$
For a target MTU of 1500 bytes: $$\text{Payload} = 1500 - 28 = 1472\text{ bytes}$$
Execute the ping probe against the destination host:
# Test for standard 1500 MTU (Payload = 1472)
ping -M do -s 1472 -c 2 198.51.100.42If the path cannot support 1500 bytes and intermediate routers handle ICMP properly, the output will state:
From 203.0.113.1 icmp_seq=1 Frag needed and DF set (mtu = 1420)However, if an intermediate router drops the packet and drops ICMP notifications (a black hole), you will see 100% packet loss with complete silence:
PING 198.51.100.42 (198.51.100.42) 1472(1500) bytes of data.
--- 198.51.100.42 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 2048msNow step down the payload to find the exact ceiling where packets pass:
# Test PPPoE MTU 1492 (Payload = 1464)
ping -M do -s 1464 -c 2 198.51.100.42
# Test VXLAN MTU 1450 (Payload = 1422)
ping -M do -s 1422 -c 2 198.51.100.42
# Test WireGuard MTU 1420 (Payload = 1392)
ping -M do -s 1392 -c 2 198.51.100.42
# Test IPsec MTU 1400 (Payload = 1372)
ping -M do -s 1372 -c 2 198.51.100.42The moment you see successful responses:
64 bytes from 198.51.100.42: icmp_seq=1 ttl=56 time=28.4 msYou have calculated the path’s bottleneck MTU: $$\text{Bottleneck MTU} = \text{Successful Payload} + 28$$
Procedure 2: Inspect Kernel Route Cache and Advertised MSS
The Linux kernel maintains routing cache entries for active destinations, tracking discovered PMTU values:
# Query the kernel routing table for a specific destination
ip route get 198.51.100.42Output on a normal, healthy route:
198.51.100.42 via 192.0.2.1 dev eth0 src 192.0.2.15 uid 1000
cacheOutput when the kernel has successfully discovered a lower PMTU via ICMP:
198.51.100.42 via 192.0.2.1 dev eth0 src 192.0.2.15 uid 1000
cache expires 582sec mtu 1420 advmss 1380Notice the mtu 1420 and advmss 1380 fields. The Linux kernel caches this discovered MTU for a duration controlled by net.ipv4.route.mtu_expires (default: 600 seconds / 10 minutes).
If a PMTU black hole exists, the mtu and advmss entries will be completely absent from the route cache output. The kernel will repeatedly try to transmit at mtu 1500, unaware that the link cannot carry it.
Procedure 3: Sniff Missing ICMP and Retransmissions with tcpdump
To verify whether your server is transmitting oversized segments and receiving silence, execute tcpdump on the egress interface:
# Sniff for ICMP Type 3 Code 4 notifications AND TCP SYN/retransmission traffic
tcpdump -nn -vv -i any 'icmp[0] == 3 and icmp[1] == 4 or (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0) or (tcp and port 443)'When a black hole occurs, tcpdump will reveal the following sequence:
# 1. Successful 3-Way Handshake
14:20:10.100124 IP 192.0.2.15.54320 > 198.51.100.42.443: Flags [S], seq 1001, win 64240, options [mss 1460,sackOK,TS val 28374912 ecr 0], length 0
14:20:10.112450 IP 198.51.100.42.443 > 192.0.2.15.54320: Flags [S.], seq 8001, ack 1002, win 65160, options [mss 1460,sackOK,TS val 9482711 ecr 28374912], length 0
14:20:10.112510 IP 192.0.2.15.54320 > 198.51.100.42.443: Flags [.], ack 1, win 64240, length 0
# 2. Client sends small TLS Client Hello (350 Bytes) - Slices through cleanly
14:20:10.113000 IP 192.0.2.15.54320 > 198.51.100.42.443: Flags [P.], seq 1:351, ack 1, win 64240, length 350
# 3. Server sends TLS Server Hello + Certificate (1500 Bytes, DF=1)
14:20:10.128000 IP 198.51.100.42.443 > 192.0.2.15.54320: Flags [.], seq 1:1461, ack 351, win 65160, length 1460
# 4. SILENCE. No ACK from client. No ICMP Type 3 Code 4 logged.
# 5. Server retransmits after Retransmission Timeout (RTO)
14:20:11.130100 IP 198.51.100.42.443 > 192.0.2.15.54320: Flags [.], seq 1:1461, ack 351, length 1460
14:20:13.140200 IP 198.51.100.42.443 > 192.0.2.15.54320: Flags [.], seq 1:1461, ack 351, length 1460
14:20:17.160400 IP 198.51.100.42.443 > 192.0.2.15.54320: Flags [.], seq 1:1461, ack 351, length 1460Notice that the server retransmits the full 1460-byte payload at 1s, 2s, 4s intervals. Because no ICMP notification arrives to inform the server of the path constraint, it never steps down its segment size.
Procedure 4: Pinpoint the TLS Stall with Timed curl Tracing
Run curl with microsecond-level trace timing to pinpoint where the connection hangs:
curl -Iv --trace-time https://api.internal.enterprise.comOutput:
14:22:01.001240 * Trying 198.51.100.42:443...
14:22:01.014520 * Connected to api.internal.enterprise.com (198.51.100.42) port 443
14:22:01.015110 * ALPN: offers h2,http/1.1
14:22:01.015890 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
[... HANGS HERE FOR 120 SECONDS ...]
14:24:01.016200 * OpenSSL SSL_connect: Connection timed out in connection to api.internal.enterprise.com:443
14:24:01.016350 * Closing connection 0
curl: (28) OpenSSL SSL_connect: Connection timed out in connection to api.internal.enterprise.com:443The freeze happens immediately after sending Client hello (1). The client is waiting for Server hello (2) and the server’s certificate chain. Because the certificate bundle exceeds the path MTU, it vanishes into the black hole.
5. Actionable Remediation Runbook
Resolving PMTU black holes requires a multi-layered approach:
- Enable Linux Kernel Packetization-Layer Path MTU Discovery (PLPMTUD) so the OS dynamically detects MTU limits without relying on ICMP.
- Implement TCP MSS Clamping on intermediate gateways, routers, and host firewalls using
iptablesandnftables. - Unblock
ICMP Type 3, Code 4on all local and perimeter firewalls.
+---------------------------------------------------------------------------------------------------+
| Multi-Tier PMTU Remediation Architecture |
+---------------------------------------------------------------------------------------------------+
| |
| [Tier 1: Linux Kernel PLPMTUD] |
| /etc/sysctl.d/99-pmtu.conf ──► net.ipv4.tcp_mtu_probing = 1 |
| (Probes MTU autonomously via TCP ACK loss; works without any ICMP) |
| |
| [Tier 2: Network / Gateway MSS Clamping] |
| iptables / nftables ────────► Clamps TCP MSS on SYN packets to interface MTU |
| (Rewrites TCP header options on the wire during handshake) |
| |
| [Tier 3: Perimeter Firewall Hardening] |
| Border & Local Firewalls ───► Explicitly permit ICMP Type 3 Code 4 (frag-needed) |
| (Restores standard RFC 1191 signaling across transit hops) |
+---------------------------------------------------------------------------------------------------+Layer 1: Enable Linux Kernel PLPMTUD (RFC 4821)
The most robust host-level fix is enabling Packetization-Layer Path MTU Discovery (PLPMTUD), specified in RFC 4821.
Unlike classic RFC 1191 PMTUD, which depends on intermediate routers generating ICMP packets, PLPMTUD operates entirely inside the TCP stack. If the kernel detects that full-sized TCP segments are continually unacknowledged while smaller segments succeed, it assumes a PMTU black hole exists and automatically steps down the connection’s MSS to a safe baseline (tcp_base_mss).
Kernel Parameters
Create or update /etc/sysctl.d/99-pmtu.conf:
# /etc/sysctl.d/99-pmtu.conf
# -------------------------------------------------------------------------
# Production Linux Kernel Path MTU Discovery & Black Hole Recovery
# -------------------------------------------------------------------------
# Controls TCP Packetization-Layer Path MTU Discovery:
# 0 = Disabled (Default in older kernels)
# 1 = Enabled when a black hole is suspected (Recommended default)
# 2 = Always enabled (Probes constantly; useful for complex tunnel networks)
net.ipv4.tcp_mtu_probing = 1
# Starting MSS used when a black hole is detected and probing begins.
# Default is 512. In modern networks, 1024 or 1280 (IPv6 min MTU) provides
# significantly higher throughput while staying below tunnel thresholds.
net.ipv4.tcp_base_mss = 1024
# Time (in seconds) the kernel caches discovered Path MTU before testing
# if a larger MTU is supported again. Default: 600s (10 minutes).
net.ipv4.route.mtu_expires = 600
# Minimum PMTU the kernel will ever accept from ICMP or probing.
# Prevents malicious attackers from forcing pathologically tiny MTUs (DoS).
net.ipv4.route.min_pmtu = 552Apply the configuration immediately:
sysctl -p /etc/sysctl.d/99-pmtu.confVerify that the runtime values took effect:
sysctl net.ipv4.tcp_mtu_probing net.ipv4.tcp_base_mssHow PLPMTUD Operates Internally
When net.ipv4.tcp_mtu_probing = 1 is active:
- TCP starts sending segments at standard MSS (e.g., 1460).
- If consecutive retransmission timeouts (RTOs) occur on full-sized segments, the kernel flags the route as a suspected black hole.
- The kernel immediately falls back to
tcp_base_mss(1024 bytes). - Since a 1024-byte payload yields a 1064-byte IP packet (well below any tunnel or PPPoE overhead), the packet passes through immediately.
- The receiver sends an ACK. The connection resumes without dropping the socket.
- The kernel then uses binary search probing to gradually increase the MSS, identifying the highest usable transmission unit for that specific path.
Layer 2: Implement TCP MSS Clamping in iptables and nftables
If you operate a router, VPN server (WireGuard, OpenVPN), Docker/Kubernetes gateway, or a Linux host with a tunnel interface, you should enforce TCP MSS Clamping.
MSS Clamping intercepts TCP SYN packets during the three-way handshake and rewrites the advertised MSS option so neither endpoint attempts to send packets larger than the intermediate link can handle.
+---------------------------------------------------------------------------------------------------+
| TCP MSS Clamping Mechanics |
+---------------------------------------------------------------------------------------------------+
| Client (MTU 1500) Gateway / Firewall (WireGuard MTU 1420) Server (MTU 1500) |
| │ │ │ |
| │ 1. TCP SYN (Advertised MSS=1460) │ │ |
| ├───────────────────────────────────►│ │ |
| │ │ *Gateway inspects SYN packet* │ |
| │ │ *Rewrites MSS: 1460 -> 1380* │ |
| │ │ │ |
| │ │ 2. Forwarded SYN (Advertised MSS=1380) |
| │ ├─────────────────────────────────────►│ |
| │ │ │ |
| │ │ 3. Server SYN-ACK (MSS=1380) │ |
| │ │◄─────────────────────────────────────┤ |
| │ │ │ |
| │ 4. Client receives SYN-ACK │ │ |
| │◄───────────────────────────────────┤ │ |
| │ |
| ▼ |
| Both endpoints now naturally limit their payloads to MSS 1380 (Packet Size = 1420 Bytes). |
| No packet ever exceeds the WireGuard tunnel MTU. Fragmentation is completely avoided. |
+---------------------------------------------------------------------------------------------------+Method A: Modern nftables Configuration (Recommended for Debian 12+, Ubuntu 22.04+, RHEL 9+)
In modern Linux distributions, nftables is the native packet-filtering engine. Add the following rule to your forward or output chain:
# Add MSS clamping to the forward chain matching the path MTU automatically
nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu
# Alternatively, clamp specifically on a VPN/tunnel egress interface (e.g., wg0)
nft add rule inet filter forward oifname "wg0" tcp flags syn tcp option maxseg size set rt mtuTo make this persistent in /etc/nftables.conf:
table inet filter {
chain forward {
type filter hook forward priority 0; policy accept;
# Clamp TCP MSS to Path MTU for all routed traffic
tcp flags syn tcp option maxseg size set rt mtu
}
}Reload nftables:
nft -f /etc/nftables.confMethod B: Legacy iptables Configuration
If your system relies on iptables (or iptables-legacy), apply clamping via the mangle table:
# Clamp MSS to Path MTU automatically for all forwarded traffic
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
# Clamp MSS for traffic originating locally and routed across a specific interface
iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o wg0 -j TCPMSS --clamp-mss-to-pmtuMethod C: Explicit Manual MSS Clamping for Known Constraints
When automatic PMTU calculation fails or when managing complex double-encapsulated tunnels (e.g., VXLAN over IPsec), explicitly force a fixed, safe MSS:
# Force MSS to 1360 bytes (safe for all IPsec, WireGuard, and PPPoE networks)
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360
iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360To make iptables rules persistent across reboots:
# On Debian / Ubuntu
apt-get install -y iptables-persistent
netfilter-persistent save
# On RHEL / AlmaLinux / Rocky Linux
iptables-save > /etc/sysconfig/iptables
systemctl enable --now iptablesLayer 3: Unblock ICMP Type 3, Code 4 on Border Firewalls
Ensure your host firewalls explicitly allow fragmentation feedback.
For iptables
Place this rule near the top of your INPUT chain, before any blanket DROP rules:
# Explicitly allow ICMP Fragmentation Needed and DF Set (Type 3, Code 4)
iptables -I INPUT 1 -p icmp --icmp-type fragmentation-needed -j ACCEPT
# For IPv6: Explicitly allow Packet Too Big (Type 2)
ip6tables -I INPUT 1 -p ipv6-icmp --icmpv6-type packet-too-big -j ACCEPTVerify rule placement:
iptables -L INPUT -n -v --line-numbers | grep -E "icmp|Chain"For nftables
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
# Accept established traffic
ct state established,related accept
# Accept required ICMP types for Path MTU Discovery
ip icmp type destination-unreachable accept
ip6 icmp type packet-too-big accept
}
}For UFW (/etc/ufw/before.rules)
Open /etc/ufw/before.rules and verify the presence of these lines under the INPUT section:
# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPTReload UFW:
ufw reloadFor CSF (/etc/csf/csf.conf)
Ensure that ICMP code 4 is permitted in ICMP_IN_TYPES:
# /etc/csf/csf.conf
# Include 3 (Destination Unreachable) and 11 (Time Exceeded)
ICMP_IN = "1"
ICMP_IN_TYPES = "0,3,4,8,11,12"Restart CSF:
csf -r6. Docker and Kubernetes Overlay Network Configuration
When container workloads experience PMTU black holes, the root cause is usually a mismatched MTU between the host physical interface and the container virtual ethernet bridge (veth).
Correcting Docker Daemon MTU
By default, Docker assigns 1500 bytes to all bridge networks. If your host VPS runs inside an OpenStack or cloud provider using a 1450-byte MTU, Docker containers will generate oversized frames that stall on egress.
Edit or create /etc/docker/daemon.json:
{
"mtu": 1450,
"bip": "172.17.0.1/16"
}Restart the Docker service:
systemctl restart dockerVerify that the docker0 interface reflects the new MTU:
ip link show docker03: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1450 qdisc noqueue state DOWN mode DEFAULT ...For existing user-defined bridge networks created via docker network create, the MTU must be set explicitly during creation:
docker network create \
--driver bridge \
--opt "com.docker.network.driver.mtu"="1450" \
production-bridgeCorrecting Kubernetes CNI MTU (Calico / Cilium)
If running Kubernetes inside cloud VMs or across WireGuard meshes, you must align the CNI overlay MTU with your underlying link budget:
- Calico: Set
veth_mtuin thecni-configConfigMap or patch the installation resource:kubectl patch installation default --type='merge' -p '{"spec":{"calicoNetwork":{"mtu":1440}}}' - Cilium: Set
tunnel: vxlanwith an explicit MTU override in the Cilium Helm values or ConfigMap:mtu: 1450
7. High-Throughput Infrastructure Architecture
While sysctl tuning and TCPMSS clamping effectively remediate PMTU black holes across mixed network paths, relying on ongoing MSS clamping introduces subtle operational overheads:
- Every TCP SYN packet must be evaluated in the kernel’s network filter pipeline (
mangletable), incurring CPU overhead at high connection rates. - Packet-mangling firewalls can disrupt end-to-end checksum offloading on virtual network interfaces (vNICs).
- In multi-tenant cloud environments, underlying hypervisors use nested Geneve/VXLAN encapsulation, variable vNIC queues, and multi-tenant virtual switches that can drop out-of-order packets or discard ICMP control messages beyond your control.
For high-throughput transactional platforms—such as high-traffic WooCommerce clusters, high-frequency payment gateways, and real-time database replication nodes—network reliability is paramount. This is where moving from virtualized instances to bare-metal infrastructure provides significant advantages.
+---------------------------------------------------------------------------------------------------+
| Cloud Overlay vs. Bare-Metal Line-Rate MTU |
+---------------------------------------------------------------------------------------------------+
| VIRTUALIZED CLOUD HYPERVISOR (Prone to PMTU Black Holes) |
| [Guest OS: MTU 1500] ──► [vNIC VirtIO] ──► [Hypervisor vSwitch] ──► [VXLAN Encapsulation: 1450] |
| Bottlenecks: Intermediate encapsulation, CPU-bound packet copying, dropped ICMP signaling. |
| |
| BARE-METAL DEDICATED HARDWARE (Deterministic Network Stack) |
| [Linux Kernel OS] ──► [Physical Intel/Mellanox 10GbE/25GbE NIC] ──► [ToR Switch Fabric] |
| Advantages: Native Layer-2 switching, unconstrained hardware queues, Jumbo Frames (MTU 9000). |
+---------------------------------------------------------------------------------------------------+Deploying on dedicated bare-metal infrastructure gives you full control over physical interface properties:
- Configure native Jumbo Frames (MTU 9000) across your private storage, Redis, and database replication backplanes, reducing per-packet CPU overhead by over 70%.
- Eliminate the hypervisor vSwitch encapsulation layer entirely, avoiding intermediate fragmentation and nested tunnel overheads.
- Avoid noisy-neighbor vNIC throttling and unpredictable cloud NAT routing behaviors.
Organizations requiring zero-compromise network stability can leverage enterprise Dedicated Servers and localized Dedicated Servers in Pakistan from Nextgen Hosting. These environments provide dedicated physical switch ports, unconstrained multi-queue NICs, enterprise Tier-1 upstream routing, and complete administrative control over the entire Layer 2 and Layer 3 network stack.
8. Summary Diagnostic & Remediation Checklist
Keep this checklist accessible during network incident escalations:
+----------------------------------------------------------------------------------------------------+
| PMTU Black Hole Quick Reference |
+----------------------------------------------------------------------------------------------------+
| |
| 1. IDENTIFY THE PROBLEM: |
| - Ping works, curl -I works, but curl with payload or TLS handshake hangs. |
| |
| 2. FIND BOTTLENECK MTU: |
| $ ping -M do -s 1472 -c 2 <target_ip> # Test 1500 MTU |
| $ ping -M do -s 1464 -c 2 <target_ip> # Test 1492 MTU (PPPoE) |
| $ ping -M do -s 1392 -c 2 <target_ip> # Test 1420 MTU (WireGuard) |
| $ ping -M do -s 1372 -c 2 <target_ip> # Test 1400 MTU (IPsec) |
| |
| 3. KERNEL AUTOMATIC MITIGATION (PLPMTUD): |
| Add to /etc/sysctl.d/99-pmtu.conf: |
| net.ipv4.tcp_mtu_probing = 1 |
| net.ipv4.tcp_base_mss = 1024 |
| Execute: sysctl -p /etc/sysctl.d/99-pmtu.conf |
| |
| 4. FIREWALL MSS CLAMPING (ROUTERS / VPNS / GATEWAYS): |
| iptables: |
| iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu |
| nftables: |
| nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu |
| |
| 5. UNBLOCK ICMP TYPE 3 CODE 4: |
| iptables -I INPUT 1 -p icmp --icmp-type fragmentation-needed -j ACCEPT |
| |
| 6. DOCKER OVERLAYS: |
| Set "mtu": 1450 in /etc/docker/daemon.json and restart docker. |
+----------------------------------------------------------------------------------------------------+By applying kernel-level MTU probing alongside proper MSS clamping and ICMP policy enforcement, systems engineers can eliminate PMTU black holes permanently, ensuring clean, predictable throughput across all network boundaries.
Eliminate Virtualization Overlays and Network Black Holes
Tired of unpredictable cloud vSwitch latency, dropped packet fragments, and tunnel encapsulation bottlenecks? Deploy your high-concurrency applications, database clusters, and payment processing APIs on Nextgen Hosting bare-metal infrastructure. Enjoy full control over physical NIC parameters, native Jumbo Frame support, and unthrottled Tier-1 connectivity backed by 24/7 senior systems engineering support.
