Resolving 'Out of Inodes' (No Space Left on Device) Errors on Linux Web Servers

As a Linux system administrator or web hosting provider, encountering the dreaded No space left on device (ENOSPC) error is a common occurrence. However, confusion arises when a quick check of disk space using df -h shows gigabytes—or even terabytes—of available storage.

If your storage isn’t full but the system refuses to create new files, your filesystem has likely fallen victim to inode exhaustion.

In this comprehensive troubleshooting guide, we will dive into what inodes are, how to pinpoint the source of inode depletion, and the most efficient commands for cleaning up millions of tiny files without locking up your server’s I/O.

Understanding Inode Exhaustion

In Linux filesystems like ext4, an inode (index node) is a data structure that stores metadata about a file, directory, or symbolic link (e.g., permissions, ownership, timestamps, and physical disk block locations). Every file or directory consumes exactly one inode.

When a filesystem is created, it is allocated a fixed number of inodes. If your web applications (such as WordPress caching plugins, PHP session managers, or mail queues) generate millions of tiny files, you can exhaust your inode pool entirely before filling up the disk’s storage blocks.

This often manifests in logs as:

[error] 28143#0: *13948332 open() "/var/lib/php/sessions/sess_a8f9b9..." failed (28: No space left on device)

or

Can't create/write to file '/tmp/#sql_4e4_0.MAI' (Errcode: 28 "No space left on device")

Step 1: Confirming the Inode Shortage

When you suspect inode exhaustion, verify it immediately using the df -i command rather than the standard df -h.

$ df -i
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
udev             998144     402   997742    1% /dev
tmpfs           1004313     635  1003678    1% /run
/dev/sda1       4194304 4194304        0  100% /
/dev/sdb1      26214400 1250000 24964400    5% /mnt/data

Notice the /dev/sda1 partition has reached 100% IUse%. This confirms our hypothesis.

Step 2: Locating the Inode Offenders

Finding the directories containing millions of files using standard commands like find or ls can take hours and severely degrade server performance.

Instead, use a highly optimized loop to count files per directory efficiently. Run this from the root directory (/) of the exhausted partition:

# Find directories containing the highest number of files/inodes
$ find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n | tail -n 10

Explanation of the pipeline:

• find . -xdev -type f: Finds all files only on the current filesystem (avoids traversing network shares or virtual mounts).
• cut -d "/" -f 2: Extracts the top-level directory names.
• uniq -c | sort -n: Counts the occurrences and sorts them numerically.

Once you identify the parent directory (e.g., /var), you can drill down further:

$ cd /var
$ find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n | tail -n 5
    4500 log
   12000 spool
 4150000 lib

Drilling into /var/lib/php/sessions, we might find millions of orphaned PHP session files that a broken cron job failed to clean up.

Step 3: Fast Deletion (Bypassing Argument list too long)

Attempting to delete millions of files with a simple rm -rf * will result in the infamous error:

-bash: /bin/rm: Argument list too long

Instead of using rm, which expands the wildcard into memory, use one of the following high-performance deletion methods.

Method A: The find + delete approach (Safe & Steady)

$ find /var/lib/php/sessions -type f -name "sess_*" -delete

This avoids the argument list limit by deleting files one by one as they are found.

Method B: The rsync trick (Blazing Fast)

If you need to instantly wipe a directory containing millions of files, the fastest method in Linux is using rsync with an empty directory.

# 1. Create an empty directory
$ mkdir /tmp/empty_dir

# 2. Sync the empty directory to the target directory (deleting the target's contents)
$ rsync -a --delete /tmp/empty_dir/ /var/lib/php/sessions/

# 3. Clean up
$ rmdir /tmp/empty_dir

Note: This approach bypasses standard system calls for file deletion in a way that minimizes CPU and I/O wait times significantly.

Preventing Inode Exhaustion at Scale

1. Fix Automated Cleanup Tasks

Ensure that systemd timers or cron jobs responsible for cleaning up temporary directories are functioning. For PHP sessions, verify the session.gc_probability and session.gc_divisor settings in php.ini.

2. Monitor Inode Usage

Update your Prometheus node_exporter or Zabbix agents to alert on inode usage crossing the 85% threshold, not just disk block usage.

3. Evaluate Your Infrastructure

In large-scale applications such as e-commerce platforms generating millions of small cache files, shared hosting environments often impose strict inode limits. Scaling vertically by migrating to Dedicated Servers or specifically targeting regional latency with Dedicated Servers in Pakistan provides unrestricted inode capacity, allowing you to format partitions with optimized inode ratios (mkfs.ext4 -i 4096) and leverage NVMe speeds required for intensive I/O operations.

Conclusion

Inode exhaustion is a classic Linux sysadmin puzzle that looks like a storage issue but is actually a metadata limit. By relying on df -i and utilizing efficient file deletion techniques like the rsync trick, you can restore service availability in minutes rather than hours. Keep your temporary directories clean, and your server will thank you.