When a Linux system operates under heavy network concurrency—such as an Nginx reverse proxy terminating 100,000 keepalive sessions, an HAProxy edge balancer routing microservices, or an Envoy gateway handling bursty API traffic—the kernel may suddenly begin dropping inbound packets and printing severe alerts to dmesg or /var/log/messages:
kernel: TCP: out of memory -- consider tuning tcp_mem
kernel: TCP: out of memory -- consider tuning tcp_memThis error is catastrophic. When the Linux networking subsystem raises this alarm, it does not simply log a passive warning. The kernel actively throttles socket buffer allocations, collapses TCP receive windows, discards incoming segments without sending resets (forcing clients to wait until timeouts expire), and forcefully terminates orphaned connections.
Many system administrators mistakenly treat this warning as a standard out-of-memory (OOM) event and attempt to restart userland applications or add swap space. However, this alert has nothing to do with the global userland virtual memory manager (vm.max_map_count or general RAM starvation). It is strictly governed by the kernel’s dedicated TCP socket memory accounting subsystem.
This diagnostic runbook deconstructs the internal mechanics of Linux TCP memory management, explains the three distinct vectors that trigger socket buffer exhaustion, provides exact procedures for inspecting kernel slab caches and socket accounting tables, and outlines a production-tested sysctl tuning baseline.
1. Anatomy of the Linux TCP Memory Subsystem
To troubleshoot TCP memory exhaustion effectively, you must understand how the Linux kernel segments network memory. Network buffers do not exist as unstructured heap allocations; they are strictly partitioned between system-wide limits, per-socket allocations, and orphan thresholds.
+-----------------------------------------------------------------------+
| Total System Physical RAM |
+-----------------------------------------------------------------------+
|
+-------------------------+-------------------------+
| |
v v
+-----------------------------+ +-----------------------------+
| Userland Applications | | Linux Kernel Space |
| (Nginx, Envoy, Redis, DBMS) | | (Page Cache, Slab Caches) |
+-----------------------------+ +-----------------------------+
|
+--------------v--------------+
| net.ipv4.tcp_mem |
| Global TCP Memory Accounting|
| [min, pressure, max] |
| ** Measured in PAGES ** |
+--------------+--------------+
|
+---------------------------------------+---------------------------------------+
| |
v v
+-----------------------------+ +-----------------------------+
| net.ipv4.tcp_rmem | | net.ipv4.tcp_wmem |
| Per-Socket RX Buffers | | Per-Socket TX Buffers |
| [min, default, max] | | [min, default, max] |
| ** Measured in BYTES ** | | ** Measured in BYTES ** |
+-----------------------------+ +-----------------------------+
| |
+---------------------------------------+---------------------------------------+
|
v
+-----------------------------+
| net.ipv4.tcp_max_orphans |
| Unattached Sockets Ceiling |
| (FIN_WAIT_1, CLOSING, etc.) |
+-----------------------------+The Critical Unit Mismatch: Pages vs. Bytes
The single most common root cause of misconfiguration is the unit of measurement used across different sysctl parameters:
net.ipv4.tcp_memis measured in kernel memory PAGES (typically 4096 bytes on x86_64).net.ipv4.tcp_rmemandnet.ipv4.tcp_wmemare measured in BYTES.
If an engineer attempts to set net.ipv4.tcp_mem using byte values—for instance, writing net.ipv4.tcp_mem = 4194304 8388608 16777216 under the assumption that these values represent bytes—the kernel will interpret the ceiling as 16,777,216 pages ($16,777,216 \times 4096 = 68.7\text{ GB}$).
Conversely, if an administrator enters byte values intended for a 64 MB buffer limit (net.ipv4.tcp_mem = 16384 32768 65536), the kernel restricts total TCP memory across the entire operating system to 65,536 pages ($268.4\text{ MB}$), causing immediate network lockups under moderate load.
Global Vector: net.ipv4.tcp_mem
The tcp_mem vector consists of three integer values:
sysctl net.ipv4.tcp_memnet.ipv4.tcp_mem = 186081 248108 372162tcp_mem[0](Min / Low): Below this page count, the kernel applies zero constraints. Even if individual sockets allocate memory aggressively up to their configured maximums, the TCP stack operates without friction.tcp_mem[1](Pressure): When total allocated TCP memory exceeds this threshold, the kernel enters TCP memory pressure mode. During pressure, socket buffer growth is curtailed, window advertisement expansion is suppressed, and aggressive packet pruning is initiated.tcp_mem[2](Max / Ceiling): The absolute hard ceiling. When total TCP page consumption reaches this number, all further socket buffer allocations fail immediately. New connections cannot allocate send/receive structures, incoming data cannot be queued, and the kernel logsTCP: out of memory -- consider tuning tcp_mem.
Per-Socket Vectors: tcp_rmem and tcp_wmem
While tcp_mem sets the global container boundary, tcp_rmem and tcp_wmem dictate the buffer boundaries of individual TCP connections:
net.ipv4.tcp_rmem = min default max(Bytes): Controls the receive buffer (SO_RCVBUF). Thedefaultvalue defines the initial buffer granted upon connection establishment. The kernel’s dynamic buffer autotuning (tcp_moderate_rcvbuf) dynamically scales this buffer up tomaxbased on the Bandwidth-Delay Product (BDP) of the connection.net.ipv4.tcp_wmem = min default max(Bytes): Controls the transmit buffer (SO_SNDBUF). Similar totcp_rmem, autotuning scales the transmit window up tomaxto fill network pipelines.
If tcp_rmem[2] (max) is configured too large (e.g., 16 MB per socket) on a system handling 50,000 concurrent sockets, a sudden surge in client latency can cause thousands of connections to expand their buffers simultaneously, exhausting the global tcp_mem[2] threshold in seconds.
2. The Three Memory Pressure Vectors
The kernel does not issue TCP: out of memory solely when global memory hits tcp_mem[2]. In the Linux kernel source code (net/ipv4/tcp_timer.c and net/ipv4/tcp_input.c), the function tcp_out_of_memory() triggers when any of the following three conditions evaluate to true.
Vector A: Global Page Exhaustion
The primary trigger occurs when:
$$\sum \text{allocated_pages}(\text{TCP}) \ge \text{tcp_mem}[2]$$
Every socket allocates memory via the kernel’s slab allocator for its underlying struct sock and incoming/outgoing sk_buff structures. In addition to the raw payload, each packet carries significant metadata overhead:
struct sk_buffheader overhead (~232 bytes on modern kernels).- Shared memory layout overhead (
skb_shared_info, ~320 bytes). - Alignment padding and slab chunk allocation rounding.
Consequently, buffering a 1 KB payload often requires a 2 KB or 4 KB slab slice. Under heavy concurrent streaming or slow-client floods, the aggregate page count across thousands of sockets breaches tcp_mem[2].
Vector B: Orphan Socket Threshold Breach
An “orphaned” TCP socket is a connection that has been closed by the userland process (e.g., Nginx issued close(fd)) but remains alive within the kernel network stack to complete the TCP teardown sequence (FIN_WAIT_1, FIN_WAIT_2, CLOSING, or LAST_ACK).
Because orphaned sockets have no userland file descriptor attached to them, their memory cannot be managed or freed by any application process. To prevent resource starvation attacks, the kernel applies two hard limits:
net.ipv4.tcp_max_orphans: The maximum absolute count of unattached sockets allowed on the host.- Orphan Memory Ratio: Even if total orphans remain below
tcp_max_orphans, if the system is under TCP memory pressure (tcp_mem[1]) and orphaned sockets consume more than a fraction of the available memory, the kernel declares an out-of-memory condition specifically for orphans.
When triggered, the kernel logs the error, increments TCPAbortOnMemory, and forcefully terminates the connection by emitting an ungraceful TCP RST segment.
Vector C: Socket Buffer Fragmentation and Receive Window Collapse
When downstream clients read data at very slow rates (e.g., mobile clients on degraded cellular networks, or deliberate Slowloris-style slow-read attacks), the application-level buffer does not drain. The kernel must store arriving segments inside sk_buff queues.
If incoming packets arrive in small chunks (e.g., 64 to 256 bytes per packet), the ratio of metadata overhead to actual payload skyrockets. The kernel’s sk_buff slab caches (skbuff_head_cache) become heavily fragmented.
When the socket reaches its receive buffer limit (tcp_rmem[2]), the kernel attempts memory pruning via tcp_prune_queue():
- It collapses adjacent
sk_buffdata structures into contiguous memory segments to eliminate metadata overhead. - If collapsing fails to recover sufficient space, it discards out-of-order packets (
ofo_queue). - If memory remains exhausted, the kernel collapses the advertised TCP receive window to 0 (
win 0), drops incoming segments, and flagstcp_out_of_memory.
3. Real-Time Diagnostic Procedures
When the TCP: out of memory alert appears, execute the following diagnostic workflow to pinpoint the exact failure vector before modifying any configuration.
Step 1: Parse /proc/net/sockstat
The single most efficient diagnostic file in Linux for network socket state is /proc/net/sockstat. Read it without userland parsing delays:
cat /proc/net/sockstatExample production output under severe load:
sockets: used 48210
TCP: inuse 34120 orphan 14502 tw 28410 alloc 35200 mem 524288
UDP: inuse 18
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0Field Interpretation Matrix
| Metric | Meaning | Critical Diagnostic Indicator |
|---|---|---|
inuse |
Active TCP sockets currently allocated by processes (ESTABLISHED, SYN_SENT, LISTEN). | Normal connection load tracking. |
orphan |
Sockets detached from userland file descriptors (FIN_WAIT_1, FIN_WAIT_2, CLOSING). |
High Risk: If close to net.ipv4.tcp_max_orphans, Vector B is actively triggering. |
tw |
Sockets in TIME_WAIT state. |
Handled by dedicated lightweight structures; does not consume standard sk_buff memory. |
alloc |
Total socket structures allocated in kernel slab memory. | Total descriptor footprint. |
mem |
Total memory consumed by the TCP subsystem, strictly measured in PAGES. | Critical: Multiply by 4096 (or system page size) to get bytes. Compare directly with tcp_mem[2]. |
In the example above, mem is 524288. Multiplying by 4096 bytes gives:
$$\frac{524288 \times 4096}{1024 \times 1024 \times 1024} = 2.0\text{ GiB}$$
Compare this value with your system’s tcp_mem setting:
sysctl net.ipv4.tcp_memnet.ipv4.tcp_mem = 262144 393216 524288Here, mem (524,288 pages) has precisely reached tcp_mem[2] (524,288 pages). Vector A is confirmed.
Step 2: Track Per-Socket Memory Footprint with ss
To find which sockets or processes are holding onto excessive buffer space, use the extended socket statistics utility ss with internal socket memory (-m) and internal TCP info (-i):
ss -t -i -m state established | head -n 20Example socket entry output:
ESTAB 0 131072 10.0.1.15:443 198.51.100.42:54210
skmem:(r0,rb131072,t0,tb262144,f4096,w0,o0,bl0,d128)
cubic wscale:7,7 rto:204 rtt:14.2/0.8 ato:40 mss:1448 rcvspace:131072 ssthresh:10 cwnd:10Decoding the skmem Metrics:
r: Receive queue byte allocation (data held in queue waiting for userland application to read).rb: Maximum receive buffer limit (SO_RCVBUF) allocated for this specific socket.t: Transmit queue byte allocation (data waiting for network ACK).tb: Maximum transmit buffer limit (SO_SNDBUF).f: Forward memory allocation (memory pre-allocated from the global pool but not yet holding packet data).w: Memory committed to write queues.o: Option memory (used for ancillary buffers, IP options, TCP filters).bl: Backlog memory allocated during connection handshake.
If rb or tb are inflated across thousands of idle connections, individual socket autotuning is over-allocating memory relative to the host’s physical capacity.
To review global socket summary statistics:
ss -sTotal: 38450
TCP: 42150 (estab 31200, closed 2100, orphaned 8400, timewait 2400)
Transport Total IP IPv6
RAW 0 0 0
UDP 14 10 4
TCP 39750 38000 1750
INET 39764 38010 1754
FRAG 0 0 0 Step 3: Inspect Kernel Memory Slab Allocations
Socket structures and packet buffers reside in the kernel’s slab caches. When memory is exhausted, check whether slab objects are leaking or failing to reclaim:
cat /proc/slabinfo | grep -E 'TCP|skbuff'Example output:
TCP 35420 35500 2176 15 8 : tunables 0 0 0 : slabdata 2366 2366 0
TCPv6 1820 1900 2304 14 8 : tunables 0 0 0 : slabdata 135 135 0
skbuff_head_cache 98420 102400 256 16 1 : tunables 0 0 0 : slabdata 6400 6400 0
skbuff_fclone_cache 42100 45000 512 16 2 : tunables 0 0 0 : slabdata 2812 2812 0To view live top slab consumers sorted by memory consumption:
slabtop -s c -o | head -n 25If skbuff_head_cache or TCP dominate active slab usage, the kernel is spending massive physical memory maintaining in-flight packet envelopes.
4. Step-by-Step Kernel Network Tuning Runbook
To eradicate TCP: out of memory errors, you must apply a balanced configuration where global limits accommodate high concurrency, per-socket buffers are restricted from ballooning excessively, and dead orphan sockets are purged aggressively.
Step 1: Calculate Safe net.ipv4.tcp_mem Values
Do not guess these values. Calculate them based on total physical memory (RAM).
Sizing Formula for High-Traffic Servers:
- Min (
tcp_mem[0]): Allocate approximately 10% of total system RAM in pages. - Pressure (
tcp_mem[1]): Allocate approximately 20% of total system RAM in pages. - Max (
tcp_mem[2]): Allocate approximately 30% of total system RAM in pages.
(Note: On dedicated network appliances or load balancers where minimal userland database memory is required, tcp_mem[2] can safely be raised to 50% of system RAM).
Example Calculation for a 64 GB RAM Host:
- Determine total pages: $$\text{Total Pages} = \frac{64 \times 1024 \times 1024 \times 1024\text{ Bytes}}{4096\text{ Bytes/Page}} = 16,777,216\text{ pages}$$
- Calculate thresholds:
- Min (10%): $16,777,216 \times 0.10 \approx 1,677,721$
- Pressure (20%): $16,777,216 \times 0.20 \approx 3,355,443$
- Max (30%): $16,777,216 \times 0.30 \approx 5,033,164$
The resulting sysctl entry:
net.ipv4.tcp_mem = 1677721 3355443 5033164Step 2: Calibrate Per-Socket Buffers (tcp_rmem and tcp_wmem)
If individual socket buffers scale too high, even a generously sized tcp_mem pool can be depleted by a relatively modest connection count.
Calculate the required buffer size using the Bandwidth-Delay Product (BDP):
$$\text{BDP} = \text{Bandwidth (bytes/sec)} \times \text{Round-Trip Time (sec)}$$
For high-concurrency reverse proxies handling hundreds of thousands of concurrent connections, set the default buffer conservatively to conserve memory, while allowing active high-throughput connections to scale up to the max boundary when needed.
# Syntax: min default max (in bytes)
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 65536 4194304- Min (4 KB): Ensures that memory-constrained connections drop to minimal overhead.
- Default (85 KB RX / 64 KB TX): Standard window size sufficient for normal HTTP/REST/Websocket payloads without over-allocating on connection start.
- Max (4 MB): High enough to saturate a 10 Gbps link over a 50ms transatlantic RTT, but low enough to prevent a single connection from consuming 16 MB or 32 MB of slab memory.
Step 3: Accelerate Orphan Socket Reclamation
Orphan socket buildup (Vector B) is the most frequent trigger of TCP out-of-memory errors on busy HTTP/API servers. When an upstream client terminates abruptly, the server’s local socket transitions to FIN_WAIT_1 or FIN_WAIT_2.
By default, Linux retries sending FIN packets up to 8 times (tcp_orphan_retries = 8), which can keep an orphaned socket stranded in kernel memory for several minutes under high packet loss.
Adjust the orphan parameters:
# Sizing total allowed orphans (scale based on RAM, e.g., 65536 to 262144)
net.ipv4.tcp_max_orphans = 131072
# Reduce orphan retries from 8 down to 1 or 2
net.ipv4.tcp_orphan_retries = 2
# Reduce lingering time in FIN-WAIT-2 state from default 60 down to 15 seconds
net.ipv4.tcp_fin_timeout = 15Step 4: Enable Dynamic Socket Buffer Moderation
Ensure dynamic receive buffer autotuning is active. This allows the kernel to moderate window growth when memory pressure begins:
net.ipv4.tcp_moderate_rcvbuf = 15. Production Configuration Template
Consolidate these settings into a dedicated sysctl configuration file at /etc/sysctl.d/99-tcp-tuning.conf.
# /etc/sysctl.d/99-tcp-tuning.conf
# Advanced TCP Socket Memory and Buffer Tuning for High-Concurrency Servers
# Calibrated for Enterprise Linux (Ubuntu, Debian, RHEL, Rocky, AlmaLinux)
# ====================================================================
# Global TCP Memory Allocation (PAGES: 1 page = 4096 bytes on x86_64)
# Values below are calibrated for a 64 GB RAM server (10%, 20%, 30%)
# ====================================================================
net.ipv4.tcp_mem = 1677721 3355443 5033164
# ====================================================================
# Per-Socket Buffer Sizes (BYTES: min default max)
# ====================================================================
# Receive buffer autotuning
net.ipv4.tcp_rmem = 4096 87380 4194304
# Transmit buffer autotuning
net.ipv4.tcp_wmem = 4096 65536 4194304
# Dynamic receive buffer auto-tuning
net.ipv4.tcp_moderate_rcvbuf = 1
# ====================================================================
# Orphan Socket Management & Teardown Acceleration
# ====================================================================
# Maximum number of unattached TCP sockets allowed
net.ipv4.tcp_max_orphans = 131072
# Orphan FIN retry attempts before hard termination (default is 8)
net.ipv4.tcp_orphan_retries = 2
# Time to hold socket in FIN-WAIT-2 before aborting (default is 60 seconds)
net.ipv4.tcp_fin_timeout = 15
# ====================================================================
# Connection Backlog and Queue Protections
# ====================================================================
# Maximum socket listen backlog for userland daemons
net.core.somaxconn = 65535
# Inbound network interface backlog queue
net.core.netdev_max_backlog = 65535
# Maximum number of sockets in TIME_WAIT state
net.ipv4.tcp_max_tw_buckets = 262144
# Protect against SYN flood attacks when queues fill
net.ipv4.tcp_syncookies = 1Applying the Changes Safely
Apply the configuration immediately without rebooting:
sysctl --systemVerify that the active kernel parameters reflect the updated configuration:
sysctl net.ipv4.tcp_mem net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.ipv4.tcp_max_orphans net.ipv4.tcp_orphan_retriesMonitor dmesg to confirm that the TCP: out of memory messages have ceased:
dmesg -T | grep -i "out of memory"6. Bare-Metal vs. Virtualized Environments for High-Concurrency Networking
While tuning kernel parameters resolves configuration-induced TCP buffer starvation, infrastructure architecture dictates the ultimate ceiling of network concurrency.
In virtualized hosting and multi-tenant hypervisors (such as typical shared or entry-level cloud VMs), virtual network interfaces (virtio-net) traverse several emulation layers:
- Packets must pass through hypervisor virtual switches (OVS or bridge drivers).
- Physical host memory is subject to dynamic memory ballooning and kernel-samepage-merging (KSM).
- vCPU scheduling contention can cause CPU steal time during sudden traffic spikes, preventing userland processes from draining receive queues quickly enough.
When a hypervisor delays servicing an Nginx or Envoy VM during a burst, socket receive buffers fill up rapidly. Even with elevated tcp_mem settings, the socket buffers accumulate unread data, tripping the kernel’s memory prune logic and causing socket drops.
For ultra-high-concurrency reverse proxies, real-time financial transaction engines, media streaming clusters, and latency-critical API gateways, deploying on bare-metal Dedicated Servers eliminates the hypervisor abstraction layer entirely. Bare-metal hardware provides:
- Unconstrained Physical RAM: Direct access to registered DDR5 memory channels without hypervisor ballooning or page sharing risks.
- Dedicated PCIe Network Controllers: Intel and Mellanox network cards with hardware-level Multi-Queue (RSS), Large Receive Offload (LRO), and SR-IOV support, allowing network packet queues to map directly to dedicated physical CPU cores.
- Deterministic Latency: Zero vCPU scheduling jitter, guaranteeing that packet queues are drained in microseconds.
For enterprises and digital platforms serving audiences across South Asia and the Middle East, hosting on Dedicated Servers in Pakistan delivers direct local peering through the Pakistan Internet Exchange (PIX), multi-gigabit upstream transit via Tier-1 national carriers, and sub-10ms round-trip times nationwide. This local network locality drastically shrinks the Bandwidth-Delay Product (BDP), reducing per-socket buffer requirements and maximizing the total concurrent connection capacity of the Linux kernel.
Diagnostic Summary Checklist
Before closing an incident related to TCP: out of memory, ensure every checkpoint below has been verified:
- Check
cat /proc/net/sockstatand determine ifmem(in 4 KB pages) has reachedtcp_mem[2]. - Verify that
tcp_memis calculated in pages rather than bytes ($1\text{ page} = 4096\text{ bytes}$). - Inspect
orphancount in/proc/net/sockstat; verify it is well belownet.ipv4.tcp_max_orphans. - Set
net.ipv4.tcp_orphan_retries = 2andnet.ipv4.tcp_fin_timeout = 15to accelerate orphan reclamation. - Verify that
tcp_rmem[2]andtcp_wmem[2]do not permit individual sockets to consume excessive buffer space under slow-client conditions. - Check
slabtopfor excessive growth inskbuff_head_cacheorTCPslab structures. - For persistent multi-gigabit workloads exceeding virtual hypervisor limits, evaluate bare-metal deployment.
Scale High-Concurrency Workloads Without Kernel Bottlenecks
Tired of hypervisor memory ballooning and network packet drops? Deploy on Nextgen's high-frequency Cloud VPS and dedicated bare-metal servers with unmetered bandwidth, enterprise NVMe storage, and 24/7 network engineering support.
Related Diagnostic Guides & Technical Runbooks
- Diagnosing and Mitigating High CPU Usage from WordPress XML-RPC Brute Force Attacks – Deep dive into web server worker exhaustion and socket backlog defense.
- Debugging Redis Object Cache Latency Spikes, TCP Backlog Drops, and Fork Freezes – Exhaustive guide to Linux TCP backlog tuning and socket latency optimization.
- Diagnosing PHP-FPM ‘server reached max_children’ and 502 Bad Gateway Timeouts – Resolving backend process exhaustion and reverse proxy connection drops.
