Troubleshooting Linux 'No space left on device': Complete Inode Exhaustion & Deleted File Handle (lsof) Runbook

Encountering the kernel error ENOSPC (No space left on device) is one of the most critical and frequently misunderstood incidents in Linux systems engineering. It halts database write operations, causes web servers to drop incoming socket connections, crashes log ingest daemons, and renders command-line shells unstable as tab completion and temporary state files fail to write to /tmp.

The immediate reaction of most systems administrators and webmasters is to execute df -h, locate the saturated partition, and execute rm -rf on large application logs. However, production environments frequently exhibit scenarios where:

1. df -h reports 100% disk utilization, but du -sh /* accounts for only a fraction of physical disk usage.
2. df -h shows tens or hundreds of gigabytes of available capacity, yet file writes, database commits, and touch commands immediately fail with touch: cannot touch 'test': No space left on device.
3. An administrator deletes a 100 GB log file via rm -f, yet the partition remains at 100% capacity with zero reclaimed bytes.

This technical runbook unpacks the low-level Virtual File System (VFS) mechanics of the Linux kernel, the difference between superblock allocation bitmaps and directory trees, unlinked open file handles, inode allocation limits, and exact, zero-downtime diagnostic and remediation commands.

---

1. Internal Filesystem Architecture: Block Allocation vs. Inode Tables

To understand why storage errors occur counterintuitively, one must examine how standard POSIX filesystems—predominantly EXT4 and XFS—structure metadata and raw block allocation.

A modern Linux filesystem separates disk storage into two distinct data structures:

• Data Blocks: The physical clusters of disk sectors where actual file contents, text, binaries, and database records reside.
• Inodes (Index Nodes): Fixed-size data structures (typically 256 bytes in EXT4) that store file metadata: file ownership (UID/GID), access permissions, file size, timestamps (atime, mtime, ctime), hard link counters, and direct/indirect block pointers indicating where the file’s data blocks are physically stored on the block device. Crucially, the file name is not stored inside the inode; file names exist only as directory entry pointers (dentry) that link a string name to an inode number.

+-----------------------------------------------------------------------------------------+
|                                    EXT4 Block Group                                     |
+-------------------+--------------------+-------------------+---------------+------------+
| Superblock Backup | Group Descriptors  | Block Bitmap      | Inode Bitmap  | Inode Table|
| (Filesystem meta) | (Block boundaries) | (Block allocation)| (Free inodes) | (Metadata) |
+-------------------+--------------------+-------------------+---------------+------------+
                                                    |                     |
                                                    v                     v
                                          Used by 'df -h'       Used by 'df -i'

The Root Cause of df vs. du Discrepancies

When troubleshooting storage, administrators invoke two primary utilities, which query completely different kernel abstractions:

• df (Disk Free): Operates at the block device level. It reads the filesystem superblock and block group descriptors directly via the statvfs() system call. The superblock maintains global counters and block allocation bitmaps representing total, free, and reserved blocks. Reading the superblock is instantaneous ($O(1)$ time complexity), regardless of how many millions of files exist on the filesystem.
• du (Disk Usage): Operates at the user-space directory tree level. It recursively traverses the directory hierarchy starting from the target path, invoking fstatat() or lstat() on every directory entry. It aggregates the block allocations referenced by visible filenames.

This structural difference produces two scenarios where df and du severely contradict one another:

1. Unlinked Open File Descriptors: When a process holds a file open while an administrator deletes it, the directory entry is removed, hiding the file from du. However, the superblock bitmap retains the blocks as allocated until the process closes the file handle.
2. Filesystem Mount Concealment: If a secondary filesystem or NFS share is mounted over a non-empty directory (e.g., mounting /dev/sdb1 to /var), the contents previously written to the underlying root partition directory become inaccessible to directory walks like du, yet the superblock of the underlying partition still registers those blocks as occupied.

---

2. The Deleted Open File Descriptor Phenomenon

The most common operational emergency on high-throughput web nodes (running Nginx, Apache, PHP-FPM, MySQL, or Java applications) is the phantom consumption of disk space caused by deleted, open file descriptors.

Kernel VFS Mechanics: Unlinking vs. Deallocating

In POSIX compliance, file deletion is a two-step operation handled by the kernel’s Virtual File System:

1. The unlink() System Call: When an operator runs rm -f /var/log/nginx/access.log, the shell invokes the unlink() syscall. The kernel removes the directory entry pointing to the inode and decrements the inode’s hard link count (i_nlink).
2. Deallocation and Block Reclamation: The kernel inspects two counters before returning the physical blocks to the filesystem free block bitmap:
   • i_nlink: The number of hard directory links pointing to the inode (must be 0).
   • i_count: The active reference counter maintained by the kernel representing open file descriptors held across all running processes.

If a daemon (such as Nginx) has opened /var/log/nginx/access.log and maintains an open file descriptor (FD 4), i_count remains greater than zero (i_count >= 1).

Administrator Action: rm -f /var/log/nginx/access.log
                     |
                     v
             unlink() Syscall
                     |
       +-------------+-------------+
       |                           |
       v                           v
Directory Entry Removed     i_nlink decremented to 0
(File invisible to 'ls'/'du')      |
                                   v
                      Kernel checks active open handles
                                   |
                     +-------------+-------------+
                     |                           |
         Process maintains open FD       No processes holding FD
                     |                           |
                     v                           v
          Data blocks REMAIN PINNED      Data blocks returned to 
          Superblock shows 100% full     free block allocation bitmap

Even though ls -la /var/log/nginx/ no longer lists the file, the filesystem blocks cannot be overwritten or released. The storage device remains at 100% capacity. If the daemon continues writing to that open descriptor, the hidden file expands indefinitely, consuming remaining blocks until complete server failure occurs.

---

3. The Inode Exhaustion Vector

A catastrophic variant of ENOSPC occurs when physical block storage is virtually empty, yet no new files can be written.

# Output showing ample physical capacity
admin@host:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  450G   42G  408G  10% /

# Immediate write failure
admin@host:~$ touch /tmp/session_test
touch: cannot touch '/tmp/session_test': No space left on device

This condition indicates Inode Exhaustion.

How Inodes Are Allocated

In the EXT4 filesystem, the total number of inodes is determined when the filesystem is formatted (mkfs.ext4) and written permanently into the superblock. By default, EXT4 allocates approximately one inode for every 16 KB of disk space (inode_ratio).

Once created, an EXT4 inode table cannot be expanded dynamically without backing up data, reformatting the block device, and restoring files. If an application generates millions of microscopic files (smaller than the 4 KB filesystem block size), every inode is claimed long before the physical storage blocks are saturated.

In XFS, inodes are allocated dynamically in 64-inode chunks as needed, but XFS still enforces a maximum inode percentage limit (controllable via maxpct, default 25% of the filesystem).

Primary Culprits of Inode Depletion

In shared hosting environments, cPanel/WHM installations, and high-concurrency microservice architectures, millions of inodes are routinely consumed by:

• PHP Session Spool Trees: Stored in /var/lib/php/sessions or /tmp. If session garbage collection cron jobs fail (e.g., misconfigured systemd timer or permissions conflicts), PHP leaves millions of sess_* zero-byte or 100-byte files lingering for months.
• Mail Transfer Agent (MTA) Queues: Unmonitored Exim or Postfix queues (/var/spool/exim/input or /var/spool/postfix/deferred) handling mail-loop bounces, spam bursts, or undeliverable transactional receipts, generating millions of queue tracking fragments.
• Node.js, Composer, and Framework Caches: Deep, nested dependency structures (node_modules, vendor, .npm, .cache) accumulating across automated deployment pipelines.
• WordPress Transients & Object Caching Artifacts: WordPress plugins configured to store transient tokens or database object cache dumps directly onto disk instead of an in-memory Redis or Memcached instance.

---

4. Real-Time Diagnostic Procedures

When an alert triggers for an ENOSPC condition, adhere to the following sequence to isolate whether the root cause is physical block saturation, an unlinked open descriptor, or inode depletion.

Step 1: Verify Block vs. Inode Capacity

Execute both commands concurrently:

df -h
df -i

Review the IUse% column. If IUse% is at 100%, skip immediately to Section 5.2. If Use% in df -h is 100% but IUse% has available headroom, check for unlinked file handles.

Step 2: Uncover Phantom Space Held by Deleted File Handles

To identify every process on the host holding references to unlinked files, invoke lsof filtered for link count zero (+L1):

lsof +L1

For large enterprise systems with thousands of open handles, sort the output by the size of the pinned file to immediately locate the offending process:

lsof +L1 -P -n | awk '{if ($7 ~ /^[0-9]+$/) print $7/1024/1024 " MB\t" $1 "\t" $2 "\t" $9 "\t" $10}' | sort -nr | head -n 15

Alternatively, query the kernel’s native /proc virtual filesystem directly without relying on external packages:

find /proc/*/fd -ls 2>/dev/null | grep -i '(deleted)' | sort -k 7 -n -r | head -n 15

Sample output:

384912  104857600 -rw-r-----   1 root   root   107374182400 Oct 26 01:14 /proc/1842/fd/4 (deleted)

In this sample, process ID 1842 holds open file descriptor 4, locking 100 GB (107374182400 bytes) of disk space that cannot be seen by directory listings.

Step 3: Pinpoint High-Density Inode Directories

If df -i reports 100%, standard tools like du are ineffective because du measures byte size rather than inode counts. Use a POSIX shell loop to crawl mount points and count directory entries without crossing filesystem boundaries (-xdev):

for d in /*; do 
  if [ -d "$d" ] && [ ! -L "$d" ]; then
    echo -n "$d: "
    find "$d" -xdev 2>/dev/null | wc -l
  fi
done | sort -n -k 2

Once the offending parent tree (such as /var) is located, drill down into its subdirectories:

find /var -maxdepth 2 -type d -exec sh -c 'echo -n "{}: "; find "{}" -xdev -type f 2>/dev/null | wc -l' \; | sort -n -k 2

Step 4: Profile Interactive Disk Usage and Live I/O Spikes

If open file descriptors and inode saturation are ruled out, track live writes and heavy storage directories:

1. Interactive Disk Profiling: Run ncdu with the -x flag to prevent scanning across mounted network shares or secondary virtual filesystems:

   ncdu -x /

2. Real-Time Write Profiling: If an unknown background daemon is aggressively filling the disk, identify the responsible process ID in real-time with iotop:

   iotop -oPa

   The -o flag limits output to processes currently doing I/O; -P groups by process; -a displays accumulated I/O consumption over time.

---

5. Zero-Downtime Remediation Runbook

When resolving an ENOSPC outage on high-concurrency production servers, restarting core daemons like MySQL or Nginx can drop thousands of active client transactions or trigger lengthy InnoDB crash recovery sequences. Use zero-downtime remediation techniques.

+-----------------------------------------------------------------------------------------+
|                                Zero-Downtime Remediation                                |
+------------------------------------+----------------------------------------------------+
| Scenario                           | Recommended Command / Strategy                     |
+------------------------------------+----------------------------------------------------+
| 100 GB log deleted, pinned by PID  | truncate -s 0 /proc/<PID>/fd/<FD>                  |
| Active log file filling partition  | : > /var/log/application.log                       |
| Million tiny files in session dir  | find /path -type f -name 'sess_*' -delete          |
| Recurring log growth               | logrotate configuration with 'copytruncate'        |
| Inode ratio starvation             | Format with 'mkfs.ext4 -i 4096' on bare-metal NVMe |
+------------------------------------+----------------------------------------------------+

5.1. Zeroing Out Pinned Deleted File Handles via /proc

When lsof +L1 reveals an active process (e.g., PID 1842) holding a 100 GB deleted file on file descriptor 4, you can reclaim the storage instantly without restarting the daemon by truncating the file directly through the Linux /proc filesystem:

# Verify the file descriptor target and current size
ls -l /proc/1842/fd/4

# Zero out the allocated blocks via truncate
truncate -s 0 /proc/1842/fd/4

Alternatively, use shell I/O redirection:

: > /proc/1842/fd/4

The null command : combined with the write-redirection operator > signals the VFS layer to truncate the file to 0 bytes. The kernel immediately reclaims all allocated data blocks and updates the superblock bitmap. df -h will reflect the reclaimed capacity instantaneously, and the process continues writing without interruption.

5.2. Purging Multi-Million-File Trees Without Shell Crashes

Attempting to delete a directory containing hundreds of thousands of files using standard wildcards results in an immediate shell failure:

admin@host:/var/lib/php/sessions$ rm -f *
-bash: /bin/rm: Argument list too long

This error (E2BIG) is enforced by the kernel’s execve() system call, which limits the total memory allocated for argument lists and environment variables (ARG_MAX, typically 2 MB).

The Solution: In-Kernel Single-Pass Deletion

Avoid pipes or argument expansion altogether. Use find with the native -delete action:

find /var/lib/php/sessions -type f -name 'sess_*' -mtime +2 -delete

The -delete action operates internally through unlinkat(), bypassing argument arrays and eliminating the overhead of launching separate child subshells.

If you must process files in batches across multiple threads, use xargs with null delimiters:

find /var/spool/exim/input -type f -print0 | xargs -0 -P 4 -n 1000 rm -f

The -P 4 flag parallelizes file unlinking across 4 CPU cores, drastically accelerating recovery on systems with fast NVMe arrays.

5.3. Hardening Logrotate Configuration

To prevent open file descriptor leaks from recurring, review your /etc/logrotate.d/ configurations. Understand the architectural trade-off between copytruncate and create:

/var/log/custom-app/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    copytruncate
}

• create (Default): Renames the existing log file and creates a new one with original permissions. For the daemon to transition to the new file, you must supply a postrotate script that signals the daemon to reopen its file handles:

  postrotate
      /usr/bin/systemctl reload nginx > /dev/null 2>/dev/null || true
  endscript

• copytruncate: Copies the active log file to a backup and truncates the original file in place (truncate -s 0). The daemon never loses its original file descriptor. While there is a microsecond window where lines written between the copy and truncate operations could be lost, it prevents descriptor lockups for third-party or legacy software that does not support SIGHUP signals.

5.4. Filesystem Format Parameters: Tuning Inode Ratios

If your application workload inherently requires hundreds of millions of small files (such as distributed caching nodes, tile rendering, or high-volume mail gateways), format the underlying partition with a customized inode ratio:

# Default EXT4 format (1 inode per 16 KB)
mkfs.ext4 /dev/nvme0n1p3

# High-density inode configuration (1 inode per 4 KB block)
mkfs.ext4 -i 4096 /dev/nvme0n1p3

# Maximum density configuration (1 inode per 2 KB)
mkfs.ext4 -i 2048 /dev/nvme0n1p3

Setting -i 4096 matches the inode creation ratio to the 4096-byte filesystem block size, ensuring that you can never run out of inodes before running out of raw physical block storage.

---

6. Infrastructure Scaling: Storage Subsystem Throughput & Virtualization Limits

When diagnosing repeated storage exhaustion incidents on busy production platforms—such as high-transaction WooCommerce shops, multi-tenant cPanel servers, and high-concurrency microservices—system administrators must evaluate the underlying hardware architecture.

On standard virtualized shared block storage, the hypervisor abstracts physical disks through a software storage fabric. When a virtual machine exhausts its disk space or inodes, administrators experience severe secondary failures:

1. Hypervisor I/O Throttling: Mass unlinking operations (e.g., executing find -delete on 5,000,000 files) generate massive metadata write IOPS. Shared cloud storage layers enforce strict IOPS and burst credit limits. When these credits deplete, the cloud platform throttles disk operations down to a fraction of normal throughput, extending a 5-minute cleanup operation into a multi-hour outage.
2. Coalesced Write Delays: Write-heavy operations like database transactions lock tables while waiting for the storage controller to commit blocks, causing high CPU wait states (%iowait) and cascading connection pool exhaustion across Nginx and PHP-FPM.

Virtualized Cloud Storage (Shared Block Layer):
+---------+     +-------------------+     +-------------------------+
| Guest VM| --> | Hypervisor Queue  | --> | Throttled SAN/EBS Fabric| (IOPS Bottleneck)
+---------+     +-------------------+     +-------------------------+

Bare-Metal Dedicated Storage (Direct PCI Express Bus):
+---------+     +-------------------+
| OS Root | --> | PCIe Gen4 NVMe    | (Direct Memory-Mapped Hardware Controller, Zero Latency)
+---------+     +-------------------+

For mission-critical production workloads, isolating your stack on enterprise hardware with dedicated non-volatile storage controllers is essential. Deploying on high-performance Dedicated Servers and localized, low-latency Dedicated Servers in Pakistan provides direct, unthrottled access to physical enterprise PCIe Gen4 NVMe arrays.

By eliminating the hypervisor abstraction layer, your systems benefit from hundreds of thousands of sustained IOPS, enabling instant inode cleanup, sub-millisecond database commits, and unconstrained I/O headroom even during massive logging events.

---

7. Automated Production Monitoring Script

Deploy this lightweight health-check script as a daily cron job or integrate it into your monitoring system (Prometheus Node Exporter, Zabbix, or Datadog) to alert your engineering team before storage limits trigger an outage.

#!/usr/bin/env bash
# /usr/local/bin/check_storage_health.sh
# Automated Inode & Phantom Open File Handle Monitor

set -euo pipefail

BLOCK_THRESHOLD=90
INODE_THRESHOLD=85
PHANTOM_SIZE_THRESHOLD_MB=1024

echo "=== Storage & Inode Health Diagnostic ==="
date -u +"%Y-%m-%d %H:%M:%SZ"

# 1. Check physical block usage
echo ""
echo "--- Checking Filesystem Block Usage (Threshold: ${BLOCK_THRESHOLD}%) ---"
df -hP -x tmpfs -x devtmpfs | awk -v thresh="$BLOCK_THRESHOLD" 'NR>1 {
    sub(/%/, "", $5);
    if ($5 >= thresh) {
        print "WARNING: Partition " $6 " is at " $5 "% capacity! (" $4 " available)";
    }
}'

# 2. Check inode allocation usage
echo ""
echo "--- Checking Filesystem Inode Allocation (Threshold: ${INODE_THRESHOLD}%) ---"
df -iP -x tmpfs -x devtmpfs | awk -v thresh="$INODE_THRESHOLD" 'NR>1 {
    sub(/%/, "", $5);
    if ($5 >= thresh) {
        print "CRITICAL: Partition " $6 " Inode usage is at " $5 "%! (" $4 " inodes free)";
    }
}'

# 3. Detect unlinked open file descriptors holding disk space
echo ""
echo "--- Checking Unlinked Deleted File Handles (> ${PHANTOM_SIZE_THRESHOLD_MB} MB) ---"
if command -v lsof >/dev/null 2>&1; then
    lsof +L1 -P -n 2>/dev/null | awk -v min_mb="$PHANTOM_SIZE_THRESHOLD_MB" 'NR>1 {
        if ($7 ~ /^[0-9]+$/) {
            mb = $7 / 1024 / 1024;
            if (mb >= min_mb) {
                printf "PID %s (%s) holds open deleted file: %s (%.2f MB)\n", $2, $1, $9, mb;
            }
        }
    }'
else
    echo "Notice: lsof not installed. Falling back to /proc inspection."
    find /proc/*/fd -ls 2>/dev/null | grep -i '(deleted)' | awk -v min_mb="$PHANTOM_SIZE_THRESHOLD_MB" '{
        mb = $7 / 1024 / 1024;
        if (mb >= min_mb) {
            printf "Descriptor %s holds unlinked file size: %.2f MB\n", $11, mb;
        }
    }'
fi

echo ""
echo "Diagnostic check completed."

Make the script executable:

chmod +x /usr/local/bin/check_storage_health.sh

---