Troubleshooting Linux File Descriptor Exhaustion: Resolving “EMFILE: Too many open files (24)” Across Nginx, PHP-FPM, MySQL & Systemd

In high-concurrency Linux server environments, few operational failures cause more sudden and widespread chaos than file descriptor exhaustion. Under heavy traffic spikes, high-frequency database read/write bursts, or unclosed socket connections, services begin dropping requests with catastrophic errors:

nginx: [alert] 14205#14205: *98102 socket() failed (24: Too many open files) while connecting to upstream
PHP Fatal error:  Uncaught Exception: fopen(/var/log/app.log): Failed to open stream: Too many open files in /var/www/html/app.php:42
[ERROR] [MY-010119] [Server] Can't open file: './production_db/orders.ibd' (errno: 24 - Too many open files)
redis-server[812]: Error allocating resources for the client: Too many open files

When this happens, web servers refuse new TLS handshakes, reverse proxies drop upstream FastCGI connections, databases fail to access .ibd tablespace files, and SSH sessions fail to fork pseudo-terminals.

The root cause often baffles system administrators because Linux employs a multi-tier hierarchy of limits. Editing /etc/security/limits.conf and restarting the daemon often does nothing because modern Linux distributions run services under systemd, which completely bypasses PAM limits.

This technical guide dissects the underlying kernel mechanics of file descriptors, establishes the diagnostic difference between EMFILE and ENFILE, and provides concrete configuration blueprints for systemd, Nginx, PHP-FPM, and MySQL/MariaDB.

---

1. The Kernel Mechanics: “Everything is a File Descriptor”

In Unix-like operating systems, the philosophy that “everything is a file” is literal at the kernel level. A file descriptor (FD) is a non-negative unsigned integer assigned by the kernel as an index into a per-process open file table.

Crucially, file descriptors are not reserved solely for static files stored on SSD or NVMe disks. In an active web stack, file descriptors are consumed by:

1. Active Network Sockets: Every inbound client HTTP/HTTPS TCP connection.
2. Upstream Network Sockets: Every connection from Nginx to PHP-FPM, Node.js, Redis, or upstream microservices.
3. Unix Domain Sockets: Local inter-process communication (IPC) such as /var/run/php-fpm.sock or /var/run/mysqld/mysqld.sock.
4. Epoll / Event Demultiplexers: Event notification descriptors (epoll_create1()) used by asynchronous event loops.
5. Pipes and FIFOs: Standard input (0), standard output (1), standard error (2), and inter-thread pipes.
6. Hardware Timers & Signals: Modern Linux kernels assign descriptors to timers (timerfd) and signals (signalfd).
7. Physical Storage Files: Static asset files, temporary cache files, SQLite databases, and log streams.

+-----------------------------------------------------------------------------------+
|                        Linux File Descriptor Hierarchy Architecture                |
+-----------------------------------------------------------------------------------+

   +-----------------------------------------------------------------------------+
   | Layer 1: Hardware & Kernel Architecture Limits                              |
   |   - /proc/sys/fs/file-max (Kernel-wide total file table ceiling)            |
   |   - /proc/sys/fs/nr_open  (Maximum allocatable FD per individual process)   |
   +-----------------------------------------------------------------------------+
                                         │
                                         ▼
   +-----------------------------------------------------------------------------+
   | Layer 2: Init System (Systemd) Process Limits                               |
   |   - Service Unit: LimitNOFILE=65535 (Controls Soft:Hard limits for daemons) |
   |   - /etc/systemd/system.conf: DefaultLimitNOFILE=                           |
   +-----------------------------------------------------------------------------+
                     │                                           │
      (Interactive Sessions Only)                 (Daemonized Services)
                     │                                           │
                     ▼                                           ▼
   +---------------------------------------+   +---------------------------------+
   | Layer 3: PAM / User Shell Limits      |   | Layer 4: Application Limits     |
   |   - /etc/security/limits.conf         |   |   - Nginx: worker_rlimit_nofile |
   |   - Applies to SSH, su, login shells  |   |   - PHP-FPM: rlimit_files       |
   |   - (Ignored by systemd services!)    |   |   - MySQL: open_files_limit     |
   +---------------------------------------+   +---------------------------------+

EMFILE vs. ENFILE: The Critical Distinction

When debugging an exhaustion event, check the exact errno or kernel syslog message to pinpoint which layer broke:

• EMFILE (Error 24: Too many open files): The process-level limit has been reached. A single process (e.g., an Nginx worker or a PHP-FPM pool child) has reached its configured soft limit (RLIMIT_NOFILE). Other processes on the operating system can still allocate file descriptors normally.
• ENFILE (Error 23: File table overflow): The system-wide kernel limit has been exhausted. The total number of open files allocated across all running processes has hit /proc/sys/fs/file-max. When ENFILE occurs, the entire operating system stalls: users cannot log in via SSH, cron jobs crash, and kernel buffers cannot allocate sockets.

---

2. Live Diagnostics: Inspecting File Descriptors in Real Time

Before modifying configuration files, evaluate current consumption and find the offending processes.

Checking System-Wide Consumption (file-nr)

Query the kernel’s active file descriptor metrics:

cat /proc/sys/fs/file-nr

Output:

34816    0    2097152

The three numbers represent:

1. Allocated file descriptors: The total number of FDs currently allocated by all processes.
2. Free allocated file descriptors: In modern Linux kernels (2.6+), this value is always 0 because allocated descriptors are freed immediately upon closure.
3. Maximum file descriptors (fs.file-max): The absolute global ceiling enforced by the kernel.

If the first number approaches the third number, you are facing an impending ENFILE system crash.

---

Inspecting Real-Time Limits of a Running Process (prlimit)

To verify the exact runtime limits applied to a specific process without restarting it, identify its PID and use prlimit:

# Find PIDs for Nginx workers
pgrep -f "nginx: worker process"

# Inspect NOFILE limits for PID 14205
prlimit --pid 14205 --nofile

Output:

RESOURCE DESCRIPTION               SOFT  HARD UNITS
NOFILE   max number of open files  1024 65535 files

Warning: If the SOFT limit is 1024, the application will throw EMFILE (24: Too many open files) as soon as the 1,025th file or socket is opened, regardless of whether the HARD limit is set to 65535 or 1048576.

You can also check the process limit table directly through the proc filesystem:

cat /proc/14205/limits | grep "Max open files"

---

Counting and Categorizing Open Descriptors per Process

To count how many descriptors an active process is currently holding:

ls -1 /proc/14205/fd | wc -l

To break down what those file descriptors actually are (sockets, disk files, pipes, epolls), pipe lsof into an aggregation pipeline:

lsof -p 14205 | awk '{print $5}' | sort | uniq -c | sort -nr

Example Output:

   842 IPv4      # Active client & upstream TCP connections
   118 unix      # FastCGI Unix domain sockets (/run/php/php8.2-fpm.sock)
    34 REG       # Open disk files (access.log, error.log, static assets)
     4 a_inode   # epoll event loop descriptors
     2 FIFO      # Pipes

---

Identifying the Top File Descriptor Consumers Across the OS

When overall descriptor consumption spikes, execute this one-liner to rank the top 10 processes consuming descriptors:

lsof -n | awk '{print $1, $2}' | sort | uniq -c | sort -nr | head -n 10

Example Output:

  12850 nginx 14205
  12410 nginx 14206
   8920 mysqld 1142
   4100 php-fpm 18940
   3980 php-fpm 18941

---

3. The Systemd Trap: Why /etc/security/limits.conf Fails for Daemons

A widespread misconception among administrators is that adding the following lines to /etc/security/limits.conf resolves file descriptor exhaustion for background services:

# /etc/security/limits.conf
*        soft    nofile   65535
*        hard    nofile   65535
nginx    soft    nofile   65535
nginx    hard    nofile   65535
mysql    soft    nofile   65535
mysql    hard    nofile   65535

Why this fails: /etc/security/limits.conf is parsed exclusively by pam_limits.so during interactive user logins (SSH, su, local console). Systemd does NOT use PAM.

When systemd forks a service unit (such as nginx.service or php8.2-fpm.service), it assigns the process limits defined in its own systemd manager configuration. By default, systemd on many Linux distributions assigns:

DefaultLimitNOFILE=1024:524288

This sets the Soft Limit to 1024 and the Hard Limit to 524288. Applications obey the soft limit unless they explicitly invoke setrlimit() in their C source code to raise it.

---

The Proper Fix: Systemd Drop-In Overrides

To permanently increase file descriptor limits for any systemd service, create a drop-in override configuration using systemctl edit:

1. Nginx Override

sudo systemctl edit nginx.service

Paste the following directive:

[Service]
LimitNOFILE=65535:65535

This creates /etc/systemd/system/nginx.service.d/override.conf and sets both the soft and hard limits to 65535.

2. PHP-FPM Override

sudo systemctl edit php8.2-fpm.service

[Service]
LimitNOFILE=65535:65535

3. MySQL / MariaDB Override

sudo systemctl edit mysqld.service
# Or for MariaDB: sudo systemctl edit mariadb.service

[Service]
LimitNOFILE=1048576:1048576

Reload and Verify

Apply the changes to the systemd manager and restart the services:

# Reload systemd configuration
sudo systemctl daemon-reload

# Restart the service
sudo systemctl restart nginx.service

# Verify active systemd properties
systemctl show nginx.service -p LimitNOFILE

Output:

LimitNOFILE=65535
LimitNOFILESoft=65535

---

4. Application-Specific Tuning & Sizing Mathematics

Raising systemd limits is necessary, but the applications themselves must be configured to utilize the available descriptor pool.

---

A. Nginx: Sizing worker_connections and worker_rlimit_nofile

In Nginx, an administrator might increase worker_connections 16384; and find that Nginx immediately crashes with EMFILE.

The Mathematical Formula

Every single proxied connection requires two file descriptors:

1. One file descriptor for the client browser connection.
2. One file descriptor for the upstream backend connection (FastCGI, proxy_pass, or uwsgi).

Furthermore, Nginx holds open descriptors for access logs, error logs, static assets served from cache, and epoll instances.

The configuration formula for worker_rlimit_nofile is:

$$\text{worker_rlimit_nofile} \ge (\text{worker_connections} \times 2) + \text{buffer (1024)}$$

Edit /etc/nginx/nginx.conf:

# /etc/nginx/nginx.conf
user www-data;
worker_processes auto;

# Must be placed in the main (global) context, NOT inside events or http
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
    use epoll;
    multi_accept on;
}

http {
    # Logging and server blocks...
}

Verify and reload Nginx:

sudo nginx -t && sudo systemctl reload nginx

Check the worker process to confirm the new limit is active:

cat /proc/$(pgrep -f "nginx: worker process" | head -n 1)/limits | grep "open files"

---

B. PHP-FPM: Tuning Pool Limits and Fixing Descriptor Leaks

If PHP-FPM workers exhaust their descriptor allocation, requests hang until the web server returns HTTP 502 Bad Gateway or HTTP 504 Gateway Timeout.

Configuring Pool Descriptor Limits

Edit the primary PHP-FPM pool configuration (e.g., /etc/php/8.2/fpm/pool.d/www.conf or /opt/cpanel/ea-php82/root/etc/php-fpm.d/www.conf):

; /etc/php/8.2/fpm/pool.d/www.conf

; Set per-process open file descriptor rlimit
rlimit_files = 65535

; Tuning process manager to prevent unclosed connection pileups
pm = dynamic
pm.max_children = 100
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 30
pm.max_requests = 1000

Why pm.max_requests matters: Setting pm.max_requests = 1000 forces PHP-FPM worker processes to respawn after handling 1,000 requests. If third-party WordPress plugins or poorly written PHP scripts fail to close file handles (fclose()), database connections, or cURL multi handles, respawning workers clears accumulated descriptor leaks.

Restart PHP-FPM:

sudo systemctl restart php8.2-fpm

---

C. MySQL & MariaDB: Aligning open_files_limit & table_open_cache

MySQL manages an internal cache of table descriptors. If you have 500 databases with 200 tables each, that represents 100,000 .ibd files. If open_files_limit is set too low, MySQL repeatedly opens and closes files, degrading I/O performance or throwing errno: 24 - Too many open files.

Edit /etc/mysql/my.cnf (or /etc/my.cnf on RHEL/cPanel):

[mysqld]
# File descriptor allocation
open_files_limit = 1048576

# Table cache tuning
table_open_cache = 8192
table_definition_cache = 4096
max_connections = 500

# InnoDB open files limit (-1 lets InnoDB open as many files as open_files_limit permits)
innodb_open_files = -1

Restart MySQL and verify inside the SQL client:

SHOW GLOBAL VARIABLES LIKE 'open_files_limit';
SHOW GLOBAL STATUS LIKE 'Open_files';
SHOW GLOBAL STATUS LIKE 'Opened_files';

If Open_files is consistently near open_files_limit, your database is bottlenecked by descriptor constraints.

When running multi-tenant database clusters, high-concurrency WooCommerce stores, or latency-critical ERP platforms, physical compute resources and bare-metal disk throughput become paramount. Deploying on high-bandwidth Dedicated Servers eliminates hypervisor-level virtio socket throttling. For South Asian enterprises requiring sub-5ms localized database queries and strict data residency, deploying on bare-metal Dedicated Servers in Pakistan provides unrestricted hardware queues and unshared kernel memory space.

---

5. Linux Kernel Tuning: Global Limits (sysctl)

Once systemd and application limits are raised, verify that the Linux kernel itself can allocate the requested descriptors.

Checking Maximum Allocatable Limit per Process (nr_open)

The kernel defines an absolute upper bound for any single process in /proc/sys/fs/nr_open (typically 1048576). You cannot set LimitNOFILE or worker_rlimit_nofile higher than nr_open.

cat /proc/sys/fs/nr_open

If you require more than 1 million descriptors per process (e.g., massive WebSocket push gateways or high-density proxy nodes), increase nr_open.

Applying Persistent Sysctl Configurations

Create /etc/sysctl.d/99-file-descriptors.conf:

# /etc/sysctl.d/99-file-descriptors.conf

# Maximum total open files across the entire operating system (ENFILE protection)
fs.file-max = 2097152

# Maximum open files any single process may request via setrlimit()
fs.nr_open = 2097152

# Epoll event queue limits
fs.epoll.max_user_watches = 1048576

# Increase local port range for outbound connections (prevents socket exhaustion)
net.ipv4.ip_local_port_range = 10240 65535

# Enable TIME_WAIT socket reuse for high-frequency upstream proxies
net.ipv4.tcp_tw_reuse = 1

Load the configuration immediately:

sudo sysctl --system

Verify that the active kernel values reflect the changes:

sysctl fs.file-max fs.nr_open

---

6. Automated Diagnostic Script: Production File Descriptor Audit

Run this diagnostic script on any Linux server to immediately identify processes nearing their soft file descriptor limits:

#!/usr/bin/env bash
# ==============================================================================
# Linux Production File Descriptor & Socket Audit Script
# ==============================================================================
set -euo pipefail

echo "================================================================="
echo " 1. SYSTEM-WIDE FILE DESCRIPTOR AUDIT"
echo "================================================================="

FILE_NR=($(cat /proc/sys/fs/file-nr))
ALLOCATED=${FILE_NR[0]}
MAX=${FILE_NR[2]}
PCT=$(( ALLOCATED * 100 / MAX ))

echo "Allocated FDs:     $ALLOCATED"
echo "Max System FDs:    $MAX (fs.file-max)"
echo "Kernel Max/Proc:   $(cat /proc/sys/fs/nr_open) (fs.nr_open)"
echo "System Usage:      $PCT%"

if [ "$PCT" -gt 85 ]; then
    echo "CRITICAL: Global file table is at $PCT% capacity! Risk of ENFILE!"
else
    echo "STATUS: Global file table headroom is healthy."
fi

echo -e "\n================================================================="
echo " 2. TOP 10 PROCESSES WITH HIGHEST DESCRIPTOR ALLOCATION"
echo "================================================================="
printf "%-8s %-15s %-10s %-10s %-8s\n" "PID" "COMMAND" "OPEN_FDS" "SOFT_LIMIT" "USAGE%"

# Iterate over running processes
for PID in $(ls -d /proc/[0-9]* | cut -d/ -f3); do
    if [ -d "/proc/$PID/fd" ] && [ -r "/proc/$PID/limits" ]; then
        OPEN_COUNT=$(ls -1 "/proc/$PID/fd" 2>/dev/null | wc -l || echo 0)
        
        # Only inspect processes with more than 50 open descriptors
        if [ "$OPEN_COUNT" -gt 50 ]; then
            SOFT_LIMIT=$(grep "Max open files" "/proc/$PID/limits" | awk '{print $4}')
            COMM=$(cat "/proc/$PID/comm" 2>/dev/null || echo "unknown")
            
            if [ "$SOFT_LIMIT" != "unlimited" ] && [ "$SOFT_LIMIT" -gt 0 ]; then
                USAGE_PCT=$(( OPEN_COUNT * 100 / SOFT_LIMIT ))
            else
                USAGE_PCT=0
            fi
            
            printf "%-8s %-15s %-10s %-10s %-8s\n" "$PID" "$COMM" "$OPEN_COUNT" "$SOFT_LIMIT" "${USAGE_PCT}%"
        fi
    fi
done | sort -k3 -nr | head -n 10

echo -e "\n================================================================="
echo " 3. CORE SERVICE SYSTEMD LIMIT VERIFICATION"
echo "================================================================="
for SVC in nginx php8.2-fpm php-fpm mysqld mariadb redis-server; do
    if systemctl is-active --quiet "$SVC" 2>/dev/null; then
        SOFT=$(systemctl show "$SVC" -p LimitNOFILESoft --value)
        HARD=$(systemctl show "$SVC" -p LimitNOFILE --value)
        echo "[ACTIVE]  $SVC -> Soft Limit: $SOFT | Hard Limit: $HARD"
    fi
done

echo -e "\nAudit Complete."

Save and run the script:

chmod +x fd_audit.sh
sudo ./fd_audit.sh

---

Summary Configuration Reference

Component	Configuration File	Key Directives	Recommended Production Values
Systemd Services	/etc/systemd/system/<service>.service.d/override.conf	LimitNOFILE=	65535:65535 (Web) / 1048576 (DB)
Nginx Main	/etc/nginx/nginx.conf	worker_rlimit_nofile worker_connections	65535 16384
PHP-FPM Pool	/etc/php/8.x/fpm/pool.d/www.conf	rlimit_files pm.max_requests	65535 1000
MySQL / MariaDB	/etc/mysql/my.cnf	open_files_limit table_open_cache	1048576 8192
Linux Kernel	/etc/sysctl.d/99-file-descriptors.conf	fs.file-max fs.nr_open	2097152 2097152

By systematically aligning kernel parameters, systemd overrides, and application-level directives, you eliminate EMFILE and ENFILE failures, allowing your infrastructure to scale cleanly during peak traffic events.