RegisterLogin
Add us as preferred source on Google →
Linux & System Troubleshooting

Troubleshooting Linux Software RAID (mdadm) Degraded Array Recovery, Rebuild I/O Starvation, and Write-Intent Bitmap Optimization Runbook

A production engineering diagnostic runbook for Linux systems administrators facing mdadm software RAID array failures, degraded states, storage I/O starvation during kernel reconstruction, and write-intent bitmap tuning.

Linux Storage & Infrastructure Reliability Team•
Troubleshooting Linux Software RAID (mdadm) Degraded Array Recovery, Rebuild I/O Starvation, and Write-Intent Bitmap Optimization Runbook

Across enterprise Linux servers, bare-metal web hosting nodes, cPanel clusters, and database clusters, the Linux kernel Multiple Device (md) driver—managed via the mdadm utility—provides native software RAID redundancy.[1][2] By mirroring (RAID 1), striping with parity (RAID 5 / RAID 6), or combining striping and mirroring (RAID 10), software RAID guards against catastrophic physical drive dropouts without necessitating proprietary hardware RAID controller locks.[2][3]

However, when a mechanical SATA, enterprise SAS, or NVMe drive degrades or fails, the resulting failure mode frequently initiates a secondary, severe outage.[1][2] When a sysadmin replaces a failed drive and triggers reconstruction, the kernel’s default resync flusher can saturate disk throughput, triggering 100% storage I/O wait (iowait).[1] Application transactions stall, MySQL/MariaDB queries exceed connection timeouts, and web servers begin returning 504 Gateway Timeouts.[1][2]

This diagnostic runbook provides systems administrators, cloud architects, and site reliability engineers with an end-to-end operational guide to detect degraded arrays, replace failed drives safely, throttle kernel resync to eliminate production starvation, and configure write-intent bitmaps.[1][2][3][4]

Failure Triage: Detecting Degraded Software RAID States

The Linux kernel continuously monitors member device health.[1] When an unrecoverable read error (URE), hardware timeout, or bad sector threshold is exceeded, the kernel ejects the offending disk partition and transitions the array into a degraded mode.[1][3]

[ Normal Operation: RAID 1 ]
Member A: /dev/sda1 [U]  <=== Synchronized
Member B: /dev/sdb1 [U]  <=== Synchronized

[ Drive Failure / Degradation Event ]
Member B: /dev/sdb1 throws I/O errors or times out
Kernel marks partition (F) and removes it from active set

[ Degraded State: [U_] or [_U] ]
Member A: /dev/sda1 [U]  <=== Operating alone (Zero redundancy)
Member B: /dev/sdb1 [_]  <=== Degraded / Missing

Diagnostic Step 1: Inspect /proc/mdstat

The fastest real-time overview of all kernel MD devices is obtained from /proc/mdstat:

cat /proc/mdstat

Degraded Array Output Example:

Personalities : [raid1] [raid10] 
md0 : active raid1 sda1[0]
      976630272 blocks super 1.2 [2/1] [U_]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

In the status string [2/1] [U_], the first number indicates total configured members (2), while the second denotes currently active members (1).[2][3] The bracketed flag [U_] confirms that /dev/sda1 is up (U), while the second drive is absent or marked faulty (_).[2]

Diagnostic Step 2: Query Detailed Array Metadata via mdadm

To inspect device UUIDs, failure counters, superblock formats, and degraded flags, query the array device:[2]

mdadm --detail /dev/md0

Key Diagnostic Metrics in Output:

  • State: clean, degraded (Confirms array is functioning without parity/mirror protection).[2]
  • Active Devices / Working Devices: Should match expected counts; a discrepancy indicates failed components.[2]
  • Failed Devices: Non-zero indicates physical ejection by the kernel.[2]
  • Events: A mismatch between superblock event counts indicates desynchronization.[2]

Diagnostic Step 3: Verify Physical Hardware Health via smartctl

Before concluding whether a drive dropped due to physical hardware failure or a transient bus reset, inspect S.M.A.R.T. telemetry using smartctl:

smartctl -H /dev/sdb
smartctl -A /dev/sdb | grep -E "Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count"
  • Current_Pending_Sector > 0: Drive has unstable sectors waiting to be remapped; physical replacement is mandatory.[4]
  • Reallocated_Sector_Ct increasing: Surface degradation in progress; immediate replacement required.[4]
  • UDMA_CRC_Error_Count increasing: Faulty SATA cable, backplane port, or SAS HBA lane rather than drive media failure.[4]

Step-by-Step Procedure: Replacing a Failed Drive

Replacing a drive in a production Linux software RAID setup requires strict adherence to sequential execution: partition table replication, array addition, bootloader reinstallation, and sync verification.[2]

[ Step 1: Eject Failed Member ]
mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm --manage /dev/md0 --remove /dev/sdb1
                    |
                    v
[ Step 2: Physical Hot-Swap / Drive Replacement ]
Physically replace /dev/sdb with identical or larger capacity enterprise drive
                    |
                    v
[ Step 3: Replicate Partition Table ]
GPT: sgdisk -R /dev/sdb /dev/sda && sgdisk -G /dev/sdb
                    |
                    v
[ Step 4: Rebuild Bootloader ]
grub-install /dev/sdb
                    |
                    v
[ Step 5: Add Partition & Begin Resync ]
mdadm --manage /dev/md0 --add /dev/sdb1

Step 1: Mark and Remove the Faulty Partition

If the kernel has not fully decoupled the failing partition, mark it as faulty and remove it from array management:[2]

# Mark partition as failed
mdadm --manage /dev/md0 --fail /dev/sdb1

# Remove partition from the array
mdadm --manage /dev/md0 --remove /dev/sdb1

If multiple arrays share partitions on /dev/sdb (e.g., /dev/md0 for /boot, /dev/md1 for swap, /dev/md2 for /), repeat the failure and removal for all partitions.[2]

Step 2: Clone Partition Tables to the Replacement Drive

Once the new physical drive is installed, its partition layout must match the healthy drive (/dev/sda) precisely.[2]

For GPT Partition Tables (Standard in Modern UEFI / NVMe Systems):

# Clone partition structure from sda to sdb
sgdisk -R /dev/sdb /dev/sda

# Randomize Disk and Partition GUIDs on sdb to prevent duplicate UUID conflicts
sgdisk -G /dev/sdb

For Legacy MBR Partition Tables:

sfdisk -d /dev/sda | sfdisk /dev/sdb

Inform the kernel of the new partition geometry without rebooting:

partprobe /dev/sdb

Step 3: Reinstall GRUB on the Replacement Drive

If the server boots from the software RAID array, the new drive must contain the bootloader. Failure to execute this step will leave the server unbootable if /dev/sda later suffers a hardware failure:[2]

grub-install /dev/sdb

Step 4: Re-Add Partitions to the Array

Add the new partitions to initiate rebuild:

mdadm --manage /dev/md0 --add /dev/sdb1

Mitigating Production Storage I/O Starvation During Resync

By default, when a drive joins an array, the Linux kernel prioritizes synchronization to minimize the vulnerability window.[1] However, this aggressiveness often exhausts disk queue depths and I/O scheduler bandwidth, freezing live applications.[1]

The kernel exposes two sysctl controls under /proc/sys/dev/raid/ to govern reconstruction throughput:[1]

  • dev.raid.speed_limit_min (Default: 1000 KiB/s in older kernels, 50000 KiB/s in modern systems): The guaranteed floor bandwidth granted to resync even when other processes demand disk access.[1]
  • dev.raid.speed_limit_max (Default: 200000 KiB/s): The ceiling bandwidth allowed when the system detects no conflicting I/O.[1]

The Production Dilemma: Starvation vs. Window of Vulnerability

Strategy speed_limit_min speed_limit_max Production Impact Risk Profile
Default Kernel Behavior 50000 KiB/s 200000 KiB/s High I/O wait during business hours; database query latency spikes Fast recovery (~2-4 hours on modern drives)
Aggressive Throttling (Daytime) 5000 KiB/s 25000 KiB/s Zero customer impact; database responses remain sub-millisecond Prolonged degraded window; risk if surviving drive fails
Uncapped Rebuild (Maintenance) 200000 KiB/s 2000000 KiB/s Maximum I/O saturation; non-rebuild tasks freeze Fastest possible recovery; recommended during scheduled downtime

Real-Time Throttling Commands

To throttle an active resync during business hours when customer traffic is peak:[1]

# Immediately drop maximum resync speed to 25 MB/s
sysctl -w dev.raid.speed_limit_max=25000
sysctl -w dev.raid.speed_limit_min=5000

Once peak business hours conclude, restore full throughput to complete synchronization overnight:[1]

# Uncap reconstruction speed to 1 GB/s for rapid completion
sysctl -w dev.raid.speed_limit_max=1000000
sysctl -w dev.raid.speed_limit_min=100000

To make speed limits persistent across reboots, add to /etc/sysctl.d/99-mdadm-raid.conf:[1]

dev.raid.speed_limit_min = 10000
dev.raid.speed_limit_max = 150000

Write-Intent Bitmaps: Eliminating Unnecessary Full Re-Syncs

One of the most powerful features of Linux mdadm is the Write-Intent Bitmap.[2]

Without a bitmap, if a server experiences an ungraceful shutdown, a momentary storage timeout, or a brief SATA bus reset, the kernel cannot know which specific data blocks were modified while the array was degraded or unsynchronized.[1][2] Consequently, it must perform a full surface scan and synchronization of the entire multi-terabyte disk, consuming hours of heavy I/O.[1][2]

With an internal write-intent bitmap, the kernel tracks dirty chunks in a dedicated superblock area.[2] When a disconnected drive reconnects, mdadm synchronizes only the specific modified chunks, reducing recovery times from 6 hours to 30 seconds.[2]

Enabling an Internal Write-Intent Bitmap Live

Bitmaps can be attached to an active array without unmounting filesystems or restarting services:[2]

mdadm --grow /dev/md0 --bitmap=internal

Verifying Bitmap Status

Check array details to confirm active bitmap tracking:

mdadm --detail /dev/md0 | grep -i bitmap

Output:

  Internal Bitmap : 2 pages [8KB], 65536KB chunk

Note on Performance Trade-off: Internal write-intent bitmaps introduce a minute metadata write overhead on random write workloads because the bitmap page must be flushed to disk before new blocks are committed.[2] For standard database, web hosting, and virtual machine workloads, this performance cost is negligible compared to the massive availability advantage of avoiding full multi-hour resyncs.[1][2]

Automated Health Monitoring and Alerting Configuration

A degraded software RAID array produces no outward errors to running web applications until a second drive fails, resulting in total data loss.[2] Immediate notification of the first drive degradation is critical.[2]

Configure mdadm daemon monitoring in /etc/mdadm/mdadm.conf (Debian/Ubuntu) or /etc/mdadm.conf (RHEL/Rocky Linux):[2]

# /etc/mdadm/mdadm.conf
MAILADDR [email protected]
MAILFROM [email protected]
PROGRAM /usr/local/bin/raid-event-alert.sh

Enable and verify the system monitoring daemon:[2]

# Update initramfs so boot-time array detection matches configuration
update-initramfs -u

# Enable and start the mdmonitor service
systemctl enable --now mdmonitor.service
systemctl status mdmonitor.service

Bare-Metal Dedicated Hardware: Why Enterprise Storage Demands Isolation

While software RAID provides robust local drive redundancy, running storage-intensive relational databases, cPanel hosting, or virtualization clusters on shared cloud virtual machines exposes infrastructure to the “noisy neighbor” phenomenon.[1] In shared cloud VPS environments, concurrent hypervisor disk rebuilds from adjacent tenants can choke underlying storage controllers, driving IOPS to zero regardless of kernel sysctl tuning.[1]

For enterprises in Pakistan demanding uncompromising I/O determinism, data sovereignty, and hardware reliability, deploying unshared bare-metal hardware is the gold standard.[1] By hosting on Nextgen’s enterprise dedicated servers and localized dedicated servers in Pakistan, your infrastructure gains:

  • Dedicated PCIe 4.0/5.0 NVMe Storage Channels: Direct CPU-attached NVMe storage bypassing hypervisor emulations, delivering sustained throughput exceeding 7,000 MB/s per drive without shared controller contention.[1]
  • Enterprise Hardware Redundancy: Enterprise hot-swap drive bays, dual redundant CRPS power supplies, and out-of-band IPMI/iKVM remote management allow physical drive replacements without system downtime.[1]
  • Sub-15ms Domestic Peering: Datacenters in Karachi and Islamabad connected to the Pakistan Internet Exchange (PIE) ensure high-volume transactional systems operate with low latency across Pakistan.[1]
Sources & Official References
  1. Linux Kernel Organization. (2026). Software RAID (md) Documentation & Sysctl Parameters (dev.raid.speed_limit_min / dev.raid.speed_limit_max). The Linux Kernel Archives. Retrieved from https://docs.kernel.org/admin-guide/md.html
  2. Michael Kerrisk & Linux Man-Pages Project. (2026). mdadm(8) — manage MD devices aka Linux Software RAID. Retrieved from https://man7.org/linux/man-pages/man8/mdadm.8.html
  3. Michael Kerrisk & Linux Man-Pages Project. (2026). md(4) — Multiple Device driver aka Linux Software RAID. Retrieved from https://man7.org/linux/man-pages/man4/md.4.html
  4. Smartmontools Open Source Project. (2026). S.M.A.R.T. Monitoring Utility (smartctl) Documentation & Error Log Analysis. Retrieved from https://www.smartmontools.org

Eliminate Storage Bottlenecks with Enterprise Dedicated Bare Metal

Protect your production databases, high-traffic web applications, and enterprise storage from drive failures and I/O starvation. Deploy Nextgen's high-performance dedicated servers equipped with enterprise NVMe RAID arrays, dedicated IPMI out-of-band management, and 24/7 hardware monitoring.

Explore Dedicated Servers Explore Pakistan Dedicated Servers

Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.

Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.