Linux Kernel TCP SYN Flood Defense: Hardening syncookies & Synack Retries

Among volumetric Denial-of-Service attack vectors, the TCP SYN Flood remains one of the most destructive and common attacks targeting enterprise infrastructure and Dedicated Servers.

The mechanics of a SYN flood exploit the fundamental three-way handshake of the TCP protocol:

1. The attacker floods the target with millions of spoofed SYN packets per second.
2. For each incoming SYN, the Linux kernel allocates an internal struct request_sock connection control block in kernel memory, places it in the SYN backlog queue, and returns a SYN-ACK.
3. The spoofed IP address never returns the final ACK.
4. The half-open connection sits in kernel memory until it times out.

Under an attack volume of just 150,000 packets per second, the kernel’s SYN backlog queue overflows in less than two seconds. Once full, the kernel prints the dreaded message to dmesg:

TCP: Possible SYN flooding on port 80. Dropping request.

At this point, legitimate users cannot establish TCP connections, resulting in connection timeouts and complete application downtime.

The native defense built into the Linux kernel is Cryptographic TCP SYN Cookies (tcp_syncookies).

When the backlog overflows, the kernel switches to a stateless mode: instead of allocating memory for half-open sockets, it encodes the connection state directly into the 32-bit Initial Sequence Number ($ISN$) of the SYN-ACK.

Here is how to configure and harden tcp_syncookies, expand kernel backlog limits, and optimize SYN-ACK retry timers to withstand massive flood attacks.

---

The Mathematics of Stateless SYN Cookies

When tcp_syncookies activates, the server generates the Initial Sequence Number ($ISN$) using a cryptographic hash:

$$ISN = \text{SHA-1}(t, \text{src_ip}, \text{src_port}, \text{dst_ip}, \text{dst_port}, \text{secret}) + s + (t \times 2^{24}) + \text{MSS Index}$$

• $t$: A 5-bit slow counter incremented every 64 seconds.
• $\text{MSS Index}$: 3 bits encoding one of 8 common Maximum Segment Size values.
• $s$: A server-side secret key known only to the kernel.

When a legitimate client responds with ACK (ack_seq = ISN + 1):

1. The kernel subtracts 1 from ack_seq to recover the original $ISN$.
2. It verifies the cryptographic hash against the 5-tuple and secret key.
3. If valid, the kernel allocates the full socket connection on-the-fly, seamlessly completing the handshake without ever having stored a half-open state!
4. Spoofed flood packets with random sequence numbers fail the hash check and are discarded with zero CPU overhead.

STATEFUL BACKLOG (Vulnerable):
Attacker SYN ──> Kernel: Allocates struct request_sock in RAM (Consumes 256 bytes)
Attacker SYN ──> Kernel: Allocates struct request_sock in RAM
Attacker SYN ──> Queue Full! Kernel drops all packets (Downtime!).

STATELESS SYN COOKIES (Immune):
Attacker SYN ──> Kernel: Backlog full? Generates cryptographic ISN in SYN-ACK
                 Kernel: Allocates 0 BYTES OF RAM!
Attacker drops   ──> Zero memory leak, zero queue exhaustion!
Legitimate ACK ──> Kernel: Verifies cryptographic ISN hash -> Establishes socket!

---

Step 1: Checking Active SYN Cookie and Backlog Status

Inspect your current kernel settings on Dedicated Servers in Pakistan:

# Query active syncookies status
sysctl net.ipv4.tcp_syncookies

# Check current SYN backlog size
sysctl net.ipv4.tcp_max_syn_backlog

# Check SYN-ACK retry limit
sysctl net.ipv4.tcp_synack_retries

---

Step 2: Hardening Kernel Parameters for SYN Flood Defense

Create /etc/sysctl.d/99-syn-defense.conf with production-hardened values:

# ====================================================================
# LINUX TCP SYN FLOOD MITIGATION & STATELESS COOKIE HARDENING
# ====================================================================

# Enable TCP SYN Cookies (1 = active when backlog fills)
net.ipv4.tcp_syncookies = 1

# Massively expand the SYN backlog queue (Default is 1024 or 2048)
# Allows absorbing larger initial bursts before falling back to syncookies
net.ipv4.tcp_max_syn_backlog = 65536

# Maximum number of pending connections queued in socket listen() backlog
net.core.somaxconn = 65536

# Reduce SYN-ACK retries from 5 to 2
# Cuts half-open socket timeout from ~63 seconds to ~9 seconds!
net.ipv4.tcp_synack_retries = 2

# Reduce initial SYN retries
net.ipv4.tcp_syn_retries = 2

# Enable TCP Timestamps (Required for 32-bit TCP Window Scale with syncookies)
net.ipv4.tcp_timestamps = 1

# Drop reset packets for invalid connections to save outbound bandwidth
net.ipv4.tcp_rfc1337 = 1

# Increase network interface packet queue backlog
net.core.netdev_max_backlog = 16384

Apply immediately to kernel memory:

sysctl -p /etc/sysctl.d/99-syn-defense.conf

---

Step 3: Why TCP Timestamps Must Remain Enabled with SYN Cookies

A common misconception among system administrators is disabling tcp_timestamps for pseudo-security.

Critical Warning: In modern Linux kernels, tcp_timestamps must be set to 1 when tcp_syncookies = 1. Because the 32-bit TCP sequence number is fully occupied by the cryptographic hash and MSS index, there are no bits left to encode TCP Window Scaling or selective acknowledgments (SACK). When timestamps are enabled, the kernel encodes the client’s window scale and SACK options inside the lower bits of the TCP Timestamp Value (TSval), preserving modern high-bandwidth TCP throughput even during an active SYN flood!

---

Step 4: Companion Nginx Backlog Configuration

Ensure that Nginx’s listen directive matches the enlarged kernel somaxconn backlog:

In /etc/nginx/nginx.conf:

server {
    listen 80 default_server backlog=65535;
    listen 443 ssl http2 default_server backlog=65535;
    # ...
}

Reload Nginx:

nginx -t && systemctl reload nginx

---

Step 5: Verification and Live Flood Telemetry

Verify that SYN cookies are operational by monitoring kernel network counters:

# Query live SYN cookie activity
nstat -az | grep -i "Syncookie"

Output:

TcpExtSyncookiesSent            128901           0.0
TcpExtSyncookiesRecv             41209           0.0
TcpExtSyncookiesFailed             142           0.0

• SyncookiesSent: Total times the kernel activated stateless cookies when backlog filled.
• SyncookiesRecv: Legitimate connections that successfully returned valid sequence numbers and connected!
• SyncookiesFailed: Malicious or forged sequence numbers rejected by the kernel.

Monitor connection state distribution during an attack:

ss -s

Even under an incoming flood of 500,000 SYN packets per second, struct request_sock memory allocation remains virtually zero, and legitimate users experience zero connection drops.

---

Defense Performance Comparison

Metric	Unhardened Defaults	Hardened SYN Defense
SYN Backlog Capacity	1,024 sockets	65,536 sockets
Half-Open Socket Timeout	~63 Seconds	~9 Seconds (-85.7%)
Memory Leak under 100k SYN/s	Kernel OOM Panic	Zero Memory Leak
Legitimate User Availability	0% (Connection Timed Out)	100% (Instant Connection)

Hardening Linux TCP SYN cookies and backlog limits provides enterprise hosting environments with impenetrable resilience against volumetric Layer-4 transport exhaustion attacks.