Troubleshooting Linux 'TCP Segmentation Offload (TSO) & Checksum Corruption': Complete ethtool NIC Runbook

In modern high-concurrency Linux environments, few network anomalies cause more persistent operational confusion than hardware offload corruption. When network interface card (NIC) silicon or virtualized hypervisor drivers miscalculate TCP checksums or truncate segment boundaries, systems manifest bizarre symptoms:

• TCP three-way handshakes (SYN, SYN-ACK, ACK) succeed instantaneously.
• Small payload requests such as ICMP echo (ping), DNS lookups, and basic HTTP headers (curl -I) return normal responses without latency.
• As soon as an application pushes a large transmission—such as a TLS Server Hello with multiple intermediate X.509 certificates, an API payload over 1,500 bytes, a MySQL query result set, or an SCP/SFTP transfer—the TCP stream hangs indefinitely, retransmits until timing out (ETIMEDOUT), or resets abruptly (ECONNRESET).

This comprehensive guide details the mechanics of TCP Segmentation Offload (TSO), Generic Segmentation Offload (GSO), Generic Receive Offload (GRO), and Hardware Checksumming (tx-checksumming/rx-checksumming). We provide step-by-step diagnostic workflows, explain the confusing tcpdump checksum paradox, and provide production runbooks for persistent driver remediation.

---

1. Internal Architecture: NIC Hardware Offloading

To sustain 10 Gbps, 25 Gbps, and 100 Gbps line rates without exhausting server CPUs, the Linux kernel offloads segment fragmentation and checksum calculations from CPU execution pipelines directly to specialized NIC ASICs.

+-----------------------------------------------------------------------+
|                         APPLICATION LAYER                             |
|         (Nginx / Node.js / PHP-FPM / MySQL / Envoy / Redis)           |
+-----------------------------------------------------------------------+
                                   |
                                   v  write() / sendmsg()
+-----------------------------------------------------------------------+
|                    LINUX KERNEL NETWORK STACK                         |
|                                                                       |
|   1. Socket Buffer (sk_buff) Allocator                                |
|      - Assembles large TCP Super-Packets (up to 64 KB / GSO Packet)   |
|      - Leaves TCP/IP Checksum Fields BLANK or Uncalculated            |
|                                                                       |
|   2. TCP Layer Logic                                                  |
|      - Bypasses MTU chunking when TSO is enabled                      |
|      - Passes super-buffers directly to NIC driver ring buffer        |
+-----------------------------------------------------------------------+
                                   |
                                   v  DMA Ring Buffer
+-----------------------------------------------------------------------+
|                    HARDWARE NIC SILICON / CONTROLLER                  |
|                                                                       |
|   1. TCP Segmentation Offload (TSO) Engine                            |
|      - Slices 64 KB super-buffer into wire-sized MTU chunks (1500 B)  |
|      - Replicates IP/TCP headers with incrementing sequence numbers   |
|                                                                       |
|   2. TX Checksum Offload Engine                                       |
|      - Calculates 16-bit 1's complement checksum per packet           |
|      - Injects checksum directly into packet headers at line rate     |
+-----------------------------------------------------------------------+
                                   |
                                   v  Physical Wire (Cat6A / Fiber)
                     [ Clean Wire-Speed Ethernet Frames ]

Key Hardware Offload Primitives

1. TSO (TCP Segmentation Offload): Allows the kernel TCP layer to construct a single giant packet (up to 65,535 bytes) and hand it via Direct Memory Access (DMA) to the NIC. The physical NIC slices this super-packet into individual MTU-sized Ethernet frames (typically 1,500 bytes) and increments TCP sequence numbers in hardware.
2. GSO (Generic Segmentation Offload): A software fallback in the Linux kernel. If the underlying NIC hardware does not support TSO, GSO delays packet segmentation until the last possible moment before handing frames to the device driver queue, minimizing CPU context switching.
3. GRO (Generic Receive Offload): The inverse of GSO on the ingress path. When multiple contiguous packets arrive from the network wire, GRO reassembles them into a single large sk_buff before passing them up to the Linux TCP/IP stack, drastically reducing per-packet kernel overhead.
4. TX/RX Checksum Offload (tx-checksumming, rx-checksumming): The CPU delegates the mathematical 16-bit one’s complement calculation of IP header and TCP payload checksums to the network controller.

---

2. Root Causes of Hardware Offload Corruption

While hardware offloading offers significant throughput gains, subtle driver, hypervisor, or firmware flaws can corrupt packets before they reach the wire.

Cause 1: Virtualized Hypervisor Driver Incompatibilities

In virtualized cloud environments (KVM virtio-net, VMware vmxnet3, Hyper-V synthetic NICs, AWS ENA, or Oracle Cloud VNICs), virtual NIC drivers pass un-checksummed buffers across the hypervisor boundary. If the host hypervisor’s virtual switch (e.g., Open vSwitch, Linux bridge, or ESXi vSwitch) fails to calculate the final checksum before passing frames to physical hardware or a VXLAN overlay, frames exit the host with an empty (0x0000) or garbage checksum.

Cause 2: Physical NIC Firmware and Driver Glitches

Budget or desktop-grade network chipsets (such as Realtek r8169, Intel e1000e, older igb, and certain Broadcom bnxt_en revisions) frequently suffer from DMA ring buffer race conditions. When high throughput bursts saturate the transmit ring buffer, offload engines can emit frames with corrupted TCP sequence numbers or bad checksums, causing upstream edge routers to silently drop packets.

Cause 3: Overlay Encapsulation (Docker Bridges & VXLAN)

When packets originate inside a Docker container or Kubernetes pod, traverse a virtual veth pair, hit a Linux bridge (docker0 or cni0), and then get encapsulated in a VXLAN/Geneve tunnel, the inner payload checksum offload flag can get lost. The outer packet receives a valid UDP checksum, but the inner TCP packet arrives at the destination with an invalid checksum.

---

3. The tcpdump False-Positive Paradox

A common source of confusion when diagnosing checksum errors is the output of tcpdump on the sender machine.

When you run tcpdump -vvv -i eth0 on the transmitting host, you will frequently observe errors like:

14:15:02.102341 IP (tos 0x0, ttl 64, id 41232, offset 0, flags [DF], proto TCP (6), length 1500)
    192.168.10.50.443 > 203.0.113.10.54322: Flags [P.], cksum 0x4f12 (incorrect -> 0xa2c1), seq 1:1449, ack 1

Important: Seeing cksum incorrect on the transmitting server does NOT necessarily mean your network is broken.

Why This Happens:

1. libpcap (the library powering tcpdump) hooks into the Linux kernel network stack at the packet filter level (AF_PACKET), which sits above the network interface card driver.
2. Because TX Checksumming is enabled, the kernel intentionally leaves the checksum blank or partially calculated, expecting the physical NIC silicon to compute it via DMA before transmission.
3. Therefore, tcpdump intercepts the raw, un-checksummed buffer and calculates what the checksum should be, concluding it is invalid.

How to Prove True Corruption:

To confirm an actual checksum offload defect, you must run packet captures on the receiving host or an intermediate switch:

# On the remote receiving server:
tcpdump -vvv -nn -i eth0 'tcp and host <sender_ip>'

If the receiving server reports cksum ... (incorrect), the frame was delivered over the wire with a corrupted checksum, causing the remote kernel’s TCP stack to silently drop the segment.

---

4. Real-Time Diagnostics and Kernel Telemetry

Follow this diagnostic workflow to identify whether offload corruption is affecting your production systems.

Step 1: Inspect Active Offload Features

Use ethtool to inspect the complete offload feature set of your network interface:

ethtool -k eth0

Sample output:

Features for eth0:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-ip-generic: off
    tx-checksum-ipv6: on
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: off
    tx-tcp-mangleid-segmentation: off
    tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off

Step 2: Check for Dropped Bad Checksum Packets

Query kernel socket statistics to see if the receiving TCP stack is dropping bad frames:

# Check cumulative TCP checksum failure counters
netstat -s | grep -i -E "checksum|bad segments"

Expected diagnostic output on a host suffering from corrupted packets:

    4,281 bad segments received
    4,281 TCP checksum errors

Using nstat to view real-time incrementing errors:

nstat -z | grep -i TcpInCsumErrors

Step 3: Inspect Kernel and Driver Ring Buffer Errors

Check dmesg for driver firmware timeouts or hardware hang reports:

dmesg -T | grep -i -E "eth0|ixgbe|virtio_net|r8169|checksum|tx_timeout|reset"

---

5. Actionable Configuration and Remediation Runbook

If you identify dropped packets or truncated TLS handshakes, systematically disable hardware offloads to isolate the faulty engine.

Method 1: Instant Non-Disruptive Toggling (Live Testing)

Disable TX and RX checksum offloading as well as TSO and GSO:

# 1. Disable TCP Segmentation Offload
sudo ethtool -K eth0 tso off

# 2. Disable Generic Segmentation Offload
sudo ethtool -K eth0 gso off

# 3. Disable Generic Receive Offload
sudo ethtool -K eth0 gro off

# 4. Disable Transmit & Receive Checksum Offloading
sudo ethtool -K eth0 tx off rx off

Verify that offloads have been disabled:

ethtool -k eth0 | grep -E "tcp-segmentation-offload|tx-checksumming|generic-receive-offload"

Once disabled, re-run your failing high-payload transfer (e.g., curl -Iv https://yourdomain.com). If the connection immediately completes without stalling, the NIC hardware offload engine or virtual driver was the primary bottleneck.

---

Method 2: Persisting Configuration Across Reboots

Because ethtool -K modifications revert upon system reboot, implement permanent rules via your distribution’s network manager.

Option A: Ubuntu / Debian (Netplan)

Edit your Netplan YAML file in /etc/netplan/01-netcfg.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      receive-checksum-offload: false
      transmit-checksum-offload: false
      tcp-segmentation-offload: false
      generic-segmentation-offload: false
      generic-receive-offload: false

Apply the configuration:

sudo netplan apply

---

Option B: systemd-networkd / udev .link Files (Universal Modern Linux)

Create /etc/systemd/network/10-eth0.link:

[Match]
OriginalName=eth0

[Link]
ReceiveChecksumOffload=no
TransmitChecksumOffload=no
TCPSegmentationOffload=no
GenericSegmentationOffload=no
GenericReceiveOffload=no

Reload udev rules:

sudo udevadm control --reload
sudo udevadm trigger

---

Option C: AlmaLinux / Rocky Linux / RHEL / cPanel (NetworkManager)

Using nmcli to disable offloads on connection profiles:

# Get active connection name
nmcli connection show

# Disable offloads on eth0
sudo nmcli connection modify "System eth0" \
  ethtool.feature-tso off \
  ethtool.feature-gso off \
  ethtool.feature-gro off \
  ethtool.feature-rx off \
  ethtool.feature-tx off

# Reactivate connection
sudo nmcli connection up "System eth0"

---

6. Infrastructure Scaling: Bare-Metal vs. Virtualized Offloads

In shared multi-tenant virtualization, multiple guest virtual machines share a physical NIC via software emulation layers (such as QEMU/KVM virtual queues). When adjacent virtual instances saturate the physical CPU or burst high-volume traffic, hypervisor CPU scheduling delays cause packet buffer timing desyncs, triggering offload checksum corruption and dropped packets.

For mission-critical production environments—including enterprise e-commerce platforms, payment gateway switches, high-concurrency database clusters, and media streaming backends—eliminating hypervisor emulation layers is essential. Deploying on bare-metal Dedicated Servers and domestic Dedicated Servers in Pakistan ensures:

• Dedicated Enterprise NIC Silicon: Physical access to high-performance Intel (e.g., E810/X520) or NVIDIA Mellanox ConnectX NIC controllers with robust hardware offload engines that do not suffer from virtual driver desyncs.
• Hardware Receive Side Scaling (RSS): Multi-queue network distribution mapping individual CPU hardware threads directly to distinct network packet queues, maintaining deterministic line-rate performance under hundreds of thousands of concurrent connections.
• Zero Hypervisor Steal Time: Guaranteed CPU execution cycles preventing buffer delays during TCP checksum calculation.
• Direct Low-Latency Peering: Sub-5ms national round-trip latency connected directly to the Pakistan Internet Exchange (PKIX) and domestic carrier-neutral routing backbones.

---

7. Summary Diagnostic Cheat Sheet

Step	Diagnostic Command	Expected Result / Action
1. Inspect Offloads	ethtool -k eth0	Check if tso, gso, and tx-checksumming are active.
2. Identify Corrupted Packets	netstat -s | grep -i checksum	If bad segments received increments, checksum corruption is occurring.
3. Live Sniffing Check	tcpdump -nn -vvv -i eth0 'tcp'	Run on the remote receiver to verify whether packets arrive with invalid checksums.
4. Instant Remediation	sudo ethtool -K eth0 tso off gso off tx off rx off	Disables hardware offload engines immediately without dropping connection state.
5. Persist Changes	Configure Netplan, systemd .link, or NetworkManager	Ensure settings survive host reboots and network interface restarts.