Filesystem corruption is one of the most stressful emergencies a system administrator can face. Whether it’s caused by abrupt power losses, underlying storage hardware failures, or memory corruption affecting the kernel buffer cache, an unmountable EXT4 partition can bring mission-critical applications to a grinding halt.
In this guide, we dive deep into diagnosing EXT4 filesystem corruption, understanding superblock failures (the infamous “bad magic number”), and safely recovering your system using backup superblocks.
Understanding the Superblock
The superblock is the core metadata record of an EXT4 filesystem. It stores critical parameters like the filesystem size, block size, inode counts, and mounting information. If the primary superblock becomes corrupted, the Linux kernel cannot recognize the filesystem, resulting in mount failures.
When attempting to mount a corrupted EXT4 partition, you’ll often encounter this dreaded error:
mount: /mnt/data: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.Checking dmesg usually confirms the superblock corruption:
$ sudo dmesg | grep -i ext4
[ 1024.134567] EXT4-fs (sdb1): VFS: Can't find ext4 filesystem
[ 1025.876543] EXT4-fs (sdb1): bad geometry: block count 134217728 exceeds size of device (67108864 blocks)
[ 1027.992213] EXT4-fs (sdb1): error loading journalStep 1: Freeze and Assess (Do Not Run fsck Immediately)
CRITICAL WARNING: Never run fsck on a mounted filesystem or without first understanding the state of the drive. If the storage device is physically failing, running fsck can cause catastrophic, irreversible data loss as it attempts to rewrite inodes.
If this is a production environment running heavy I/O workloads, such as a large database, consider your infrastructure. Shared cloud block storage can sometimes encounter IOPS bottlenecks or latency spikes leading to partial journaling writes. To bypass shared IOPS limitations and filesystem corruption issues on heavy I/O applications, many enterprises migrate heavy database writes to hardware RAID on bare-metal Dedicated Servers or Dedicated Servers in Pakistan.
Before proceeding, take a bit-for-bit backup of the failing block device using ddrescue:
sudo ddrescue -d -r3 /dev/sdb1 /path/to/backup.img /path/to/rescue.logStep 2: Locating Backup Superblocks
EXT4 is designed with resilience in mind. It creates multiple backup copies of the superblock distributed throughout the block groups.
To find the locations of these backup superblocks, we use the mke2fs command in dry-run mode (-n). Do not forget the -n flag, or you will accidentally format the partition!
sudo mke2fs -n /dev/sdb1Output:
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 67108864 4k blocks and 16777216 inodes
Filesystem UUID: a1b2c3d4-e5f6-4a1b-9c2d-3e4f5a6b7c8d
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872The numbers at the bottom are the exact block locations of our backup superblocks.
Step 3: Verifying and Repairing the Filesystem
Now that we have the backup superblock locations, we can use e2fsck to point the recovery tool to an alternate superblock.
First, do a dry-run read-only check to see if the alternate superblock is valid and intact:
sudo e2fsck -b 32768 -n /dev/sdb1If you see output indicating that it recognizes the filesystem and starts listing inode corrections (instead of instantly throwing a bad magic number error), the superblock is valid.
Once confirmed, remove the -n flag and run the actual repair with the -y flag to automatically answer “yes” to repair prompts:
sudo e2fsck -b 32768 -y /dev/sdb1Log Snippet of a Successful Repair:
e2fsck 1.45.6 (20-Mar-2020)
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** ext3 journal has been deleted - filesystem is now ext2 only ***
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #0 (32254, counted=32253).
Fix<y>? yes
Free inodes count wrong for group #0 (8181, counted=8182).
Fix<y>? yes
/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 125345/16777216 files (0.5% non-contiguous), 4523412/67108864 blocksStep 4: Recreating the Journal (If Dropped)
In the output above, the journal was corrupted and successfully deleted by e2fsck, effectively converting the filesystem back to ext2. We must recreate the EXT4 journal before mounting.
Use tune2fs to add a new journal:
sudo tune2fs -j /dev/sdb1Finally, try mounting the partition again:
sudo mount /dev/sdb1 /mnt/dataVerify everything is intact:
df -h /mnt/data
ls -la /mnt/data/lost+found/Any orphaned files or fragmented data streams recovered during the e2fsck process will be placed in the lost+found directory, labeled by their inode number.
Conclusion
Superblock corruption does not automatically mean total data loss. By carefully imaging the drive, locating backup superblocks via mke2fs -n, and strategically running e2fsck, you can often recover unmountable EXT4 filesystems with minimal downtime.
