Troubleshooting Linux Kernel Soft Lockup, Hard Lockup, NMI Watchdog Stalls, and RCU Grace Period Freezes Runbook

Diagnose and resolve 'watchdog: BUG: soft lockup - CPU# stuck', hard lockups, NMI watchdog stalls, and rcu_sched grace period freezes on production Linux servers. Step-by-step diagnostic runbook covering kernel sysctl parameters, crash dump analysis, and spinlock contention.

Troubleshooting Linux Kernel Soft Lockup, Hard Lockup, NMI Watchdog Stalls, and RCU Grace Period Freezes Runbook

Production Linux servers powering high-concurrency relational databases, virtualization hypervisors, and distributed storage nodes occasionally suffer from sudden system unresponsiveness, erratic latency spikes, or kernel panics. During post-incident analysis, system administrators frequently find alarming traces inside the kernel ring buffer (dmesg), such as watchdog: BUG: soft lockup - CPU#4 stuck for 22s! or rcu: INFO: rcu_sched self-detected stall on CPU.[1]

These kernel alarms indicate that a logical CPU core is stuck executing kernel-space code without yielding execution to the OS scheduler or processing hardware interrupts within the configured watchdog threshold. If left unmitigated, watchdog events trigger unpredictable cascading timeouts across TCP sockets, cluster heartbeats (Corosync/Pacemaker), and active database transactions. This runbook provides production-tested diagnostic commands, architectural explanations of lockup mechanisms, crash log dissection, and kernel parameter tuning to eliminate soft and hard lockups across bare-metal and enterprise virtualized infrastructure.


1. Architectural Anatomy of Linux Kernel Watchdogs

The Linux kernel incorporates three distinct internal watchdog mechanisms designed to monitor CPU responsiveness, prevent deadlock conditions, and force automated recovery when cores become completely wedged:

Watchdog Subsystem Detection Method Interrupt State Default Trigger Interval Kernel Panic Control Parameter
Soft Lockup Watchdog Per-CPU watchdog/N kernel thread (SCHED_FIFO) Local IRQs Enabled $2 \times \text{watchdog_thresh}$ (20s) kernel.softlockup_panic
Hard Lockup Watchdog Non-Maskable Interrupts (NMI) via APIC / perf counters Local IRQs Disabled $1 \times \text{watchdog_thresh}$ (10s) kernel.hardlockup_panic
RCU Stall Detector Periodic RCU GP timer checking quiescent states Either Enabled or Disabled rcu_cpu_stall_timeout (21s) kernel.panic_on_rcu_stall

Soft Lockup Mechanism

A soft lockup occurs when a task executes inside kernel mode (such as inside a system call, memory allocation routine, or driver handler) for longer than twice the kernel.watchdog_thresh interval without yielding or calling schedule().[2]

Because local CPU interrupts remain enabled, timer interrupts continue to fire. A high-priority real-time kernel thread, watchdog/N (configured with scheduling policy SCHED_FIFO and priority 99), is woken periodically by a timer interrupt to touch a per-CPU timestamp (watchdog_touch_ts). A secondary high-resolution timer (hrtimer) monitors this timestamp. If the timestamp is older than $2 \times \text{watchdog_thresh}$ seconds, the kernel detects that the watchdog/N thread has been starved from execution and logs watchdog: BUG: soft lockup.

Hard Lockup Mechanism

A hard lockup occurs when a CPU is looping inside kernel space with local hardware interrupts disabled (e.g., via local_irq_disable() or inside a nested spin_lock_irqsave() critical section). Because local interrupts are masked, standard timer interrupts cannot fire, rendering the soft lockup thread useless.[3]

To catch hard lockups, the Linux kernel leverages the Local Advanced Programmable Interrupt Controller (APIC) and hardware performance monitoring units (PMU) via the perf subsystem to generate periodic Non-Maskable Interrupts (NMIs). Since NMIs cannot be masked by standard CPU interrupt flags, the NMI handler interrupts the stuck core, evaluates the cycle counters, and detects that the timer interrupt handler has stopped incrementing its tick counter (hrtimer_interrupts). When this occurs, the NMI watchdog emits watchdog: Watchdog detected hard LOCKUP on cpu.

RCU Grace Period Stalls

Read-Copy Update (RCU) is a scalable synchronization mechanism ubiquitous in the Linux kernel networking and VFS layers. Readers traverse data structures without acquiring expensive mutexes by disabling preemption inside rcu_read_lock() and rcu_read_unlock() blocks. Writers modify pointers atomically and defer the freeing of old memory until all existing readers have passed through a “quiescent state” (a context switch or CPU idle transition), completing an RCU grace period.[4]

If a CPU or task remains inside an RCU read-side critical section or fails to yield for longer than rcu_cpu_stall_timeout seconds, memory reclamation halts across the kernel, and the RCU stall detector logs rcu: INFO: rcu_sched self-detected stall on CPU.


2. Decrypting Kernel Stack Traces in dmesg

When a watchdog fires, the kernel dumps a detailed diagnostic report to dmesg and /var/log/messages. Analyzing the instruction pointer, call trace, and task context reveals the precise operational bottleneck.

Case 1: Soft Lockup with Memory Compaction Contention

[14285.104821] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [mysqld:32145]
[14285.104828] Modules linked in: xfs ext4 dm_mod kvm_intel kvm irqbypass crct10dif_pclmul
[14285.104845] CPU: 4 PID: 32145 Comm: mysqld Kdump: loaded Tainted: G             L  6.6.0-14-generic
[14285.104850] RIP: 0010:native_queued_spin_lock_slowpath+0x82/0x2a0
[14285.104855] Code: 48 89 f0 48 c1 e8 10 66 31 c0 f3 90 8b 07 66 85 c0 75 f7 8b 07
[14285.104860] RSP: 0018:ffffa8b48879b9d8 EFLAGS: 00000206
[14285.104863] RAX: 0000000000000101 RBX: ffff9a1c7c2b4e00 RCX: 0000000000000000
[14285.104866] RDX: 0000000000000000 RSI: 0000000000000101 RDI: ffff9a1c7c2b4e18
[14285.104870] Call Trace:
[14285.104872]  <TASK>
[14285.104875]  _raw_spin_lock+0x24/0x30
[14285.104878]  compact_zone+0x4c2/0x890
[14285.104882]  try_to_compact_pages+0x138/0x2c0
[14285.104886]  __alloc_pages_direct_compact+0x85/0x1e0
[14285.104890]  __alloc_pages_slowpath.constprop.0+0x412/0xba0
[14285.104894]  __alloc_pages+0x21b/0x2a0
[14285.104898]  alloc_pages_vma+0x8a/0x1f0
[14285.104902]  do_huge_pmd_anonymous_page+0x14f/0x7b0
[14285.104906]  __handle_mm_fault+0x8b2/0xe20
[14285.104910]  handle_mm_fault+0xc5/0x280
[14285.104914]  do_user_addr_fault+0x21e/0x650
[14285.104918]  exc_page_fault+0x7b/0x170
[14285.104922]  asm_exc_page_fault+0x27/0x30
[14285.104926]  </TASK>

Diagnostic Interpretation:

  • Triggering Process: mysqld (PID 32145) running on CPU 4.
  • Fault Location (RIP): native_queued_spin_lock_slowpath. The process is spinning on a kernel lock.
  • Call Trace Sequence: An anonymous page fault triggered do_huge_pmd_anonymous_page attempting to allocate Transparent Huge Pages (THP). Because memory is fragmented, the kernel invoked try_to_compact_pages and entered compact_zone. Multiple CPU cores attempted to acquire zone locks simultaneously, inducing extreme spinlock contention and starving the CPU for 22 seconds.

Case 2: Hard Lockup in Hardware Interrupt Handler

[89214.301211] watchdog: Watchdog detected hard LOCKUP on cpu 2
[89214.301215] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.6.0-14-generic
[89214.301218] RIP: 0010:megasas_complete_cmd_dpc+0x15a/0x320 [megaraid_sas]
[89214.301225] Call Trace:
[89214.301227]  <NMI>
[89214.301230]  watchdog_hardlockup_check+0x88/0x110
[89214.301234]  perf_event_nmi_handler+0x28/0x50
[89214.301238]  nmi_handle+0x62/0x160
[89214.301242]  default_do_nmi+0x54/0x120
[89214.301246]  exc_nmi+0xb8/0xe0
[89214.301249]  asm_exc_nmi+0x31/0x40
[89214.301253]  <<INTERRUPT>>
[89214.301256]  _raw_spin_lock_irqsave+0x32/0x40
[89214.301260]  megasas_complete_cmd_dpc+0x78/0x320 [megaraid_sas]
[89214.301264]  tasklet_action_common.constprop.0+0xad/0x120
[89214.301268]  handle_softirqs+0xdf/0x2c0
[89214.301272]  irq_exit_rcu+0x89/0xb0

Diagnostic Interpretation:

  • Triggering Component: SAS hardware RAID driver (megaraid_sas) servicing pending I/O completion frames.
  • Failure Point: CPU 2 acquired a lock with _raw_spin_lock_irqsave, disabling interrupts. The driver fell into an unrecoverable busy-loop waiting for a frozen controller register response, preventing interrupts from resuming for longer than 10 seconds. The hardware NMI watchdog fired and caught the lockup.

3. Root Cause Taxonomy of Production Kernel Lockups

Understanding the underlying operational conditions that lead to soft and hard lockups prevents misdirected troubleshooting:

1. Memory Compaction and Direct Reclaim Exhaustion

When physical memory allocation headroom is exhausted and memory is heavily fragmented, kernel tasks requiring high-order allocations (such as Transparent Huge Pages or jumbo network frames) enter direct reclaim and zone compaction. If many worker threads simultaneously loop in compact_zone() acquiring zone->lock, the system enters lock starvation, triggering soft lockup warnings.

2. Hypervisor vCPU Scheduling Inversion (Steal Time)

In cloud instances and virtualized VPS environments with overcommitted hypervisor hosts, the host scheduler may preempt a virtual CPU (vCPU) while that vCPU is holding an in-kernel spinlock inside the guest operating system. When other guest vCPUs attempt to acquire the same spinlock, they spin endlessly waiting for the descheduled vCPU to run again. If hypervisor steal time exceeds 20 seconds, the guest kernel logs a soft lockup.

3. Disk I/O Stalls and Dirty Page Flush Locks

When write-heavy workloads flood the page cache faster than underlying NVMe or SAS controllers can commit data to non-volatile flash, the kernel throttles dirty page generation. Tasks attempting disk writes get blocked on inode locks (ext4_writepages, xfs_vm_writepage) or dm-crypt encryption queues, accumulating long lockup intervals.

4. Broken Driver Loops and Hardware Controller Freezes

Faulty firmware in network adapters (e.g., ancient Realtek or Mellanox driver firmwares) or failing PCIe devices that stop acknowledging interrupts force kernel driver interrupt handlers into infinite polling loops with interrupts disabled, resulting in hard lockups.


4. Live Diagnostic and Forensic Runbook

When diagnosing an active or past lockup incident, follow this structured diagnostic checklist:

Step 1: Scan the Kernel Log for Watchdog Signatures

Execute a timestamped search across the kernel ring buffer and syslog archives:

# Search current kernel ring buffer for all watchdog, RCU, and lockup warnings
sudo dmesg -T | grep -E -i "soft lockup|hard lockup|rcu_sched|watchdog|blocked for more than"

# Search historical persistent journal logs
sudo journalctl -k --since "24 hours ago" | grep -E -i "soft lockup|hard lockup|rcu_sched|hung_task"

Step 2: Inspect Current Kernel Watchdog Configuration

Verify the active watchdog parameters and status across the running system:

# Query all active watchdog parameters via sysctl
sudo sysctl -a | grep -E "kernel\.(watchdog|softlockup|hardlockup|nmi_watchdog|panic_on)"

Expected standard production baseline:

kernel.hardlockup_all_cpu_backtrace = 0
kernel.hardlockup_panic = 0
kernel.nmi_watchdog = 1
kernel.panic_on_rcu_stall = 0
kernel.soft_watchdog = 1
kernel.softlockup_all_cpu_backtrace = 0
kernel.softlockup_panic = 0
kernel.watchdog = 1
kernel.watchdog_thresh = 10

Step 3: Monitor CPU Interrupts and Steal Time

Determine whether CPU cores are suffering from hardware IRQ imbalances or hypervisor preemption:

# Monitor per-CPU interrupts every 2 seconds
watch -n 2 -d "cat /proc/interrupts | head -n 30"

# Check CPU steal time using mpstat (sysstat package)
mpstat -P ALL 2 5

Pay special attention to the %steal column. If %steal spikes above 10% on virtualized instances, the root cause is external host hypervisor contention rather than guest software failure.

Step 4: Trace Kernel Lock Contention with perf

To pinpoint which kernel functions are consuming cycles spinning on locks during heavy load:

# Record kernel call traces for lock contention analysis
sudo perf top --sort comm,dso,symbol -e cycles:k

Look for symbols such as native_queued_spin_lock_slowpath, _raw_spin_lock, or mutex_spin_on_owner. If these dominate the top entries, locate the calling subsystem using the interactive perf report call-graph trace.


5. Kernel Tuning & Production Hardening Runbook

Implement the following production-tested configuration adjustments to mitigate soft lockups, prevent spurious panics, and optimize memory allocation performance.

1. Tune Watchdog Threshold for Heavy Server Workloads

On high-throughput bare-metal servers, large multi-socket NUMA database nodes, or virtualized hosts where transient load bursts can cause momentary scheduling delays, raising watchdog_thresh prevents false-positive panics:[5]

# Increase the watchdog threshold from 10 to 20 seconds
# Soft lockups will trigger after 40 seconds instead of 20 seconds
sudo sysctl -w kernel.watchdog_thresh=20

# Persist configuration across reboots
sudo bash -c 'cat <<EOF > /etc/sysctl.d/99-kernel-watchdog.conf
# Extend kernel watchdog threshold to absorb transient I/O spikes
kernel.watchdog_thresh = 20

# Avoid automated kernel panic on non-fatal soft lockups
kernel.softlockup_panic = 0

# Prevent panic on RCU stalls
kernel.panic_on_rcu_stall = 0

# Enable all CPU backtraces to capture full system state during a lockup
kernel.softlockup_all_cpu_backtrace = 1
kernel.hardlockup_all_cpu_backtrace = 1
EOF'

# Reload sysctl configurations
sudo sysctl --system

2. Mitigate Transparent Huge Pages (THP) Compaction Freezes

Memory compaction stalls during THP allocations represent one of the most common causes of mysqld, postgres, and java soft lockups. Reconfigure THP defragmentation from aggressive synchronous mode (always) to asynchronous background compaction (madvise or defer):

# Check current THP defragmentation mode
cat /sys/kernel/mm/transparent_hugepage/defrag

# Reconfigure THP defragmentation to avoid synchronous stall locks
echo "madvise" | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# For database servers (MySQL, MariaDB, PostgreSQL), disable THP completely:
echo "never" | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo "never" | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# Persist via systemd or GRUB boot options (/etc/default/grub)
# Add "transparent_hugepage=never" to GRUB_CMDLINE_LINUX_DEFAULT

3. Tune Virtual Memory Dirty Ratios to Prevent Storage Flush Hangs

When write bursts overwhelm storage controllers, the kernel blocks write system calls until dirty pages are flushed to disk. Adjusting dirty ratios forces the kernel to write data continuously in smaller background batches rather than executing huge, blocking synchronous flushes:

sudo bash -c 'cat <<EOF > /etc/sysctl.d/99-vm-dirty-tuning.conf
# Start background flushing when dirty memory hits 5% of RAM
vm.dirty_background_ratio = 5

# Block write operations only when dirty memory reaches 15% of RAM
vm.dirty_ratio = 15

# Reduce the age threshold for dirty pages to be written out (in centiseconds)
vm.dirty_expire_centisecs = 1500
vm.dirty_writeback_centisecs = 500
EOF'

sudo sysctl --system

4. Enable Automated Crash Dumps (kdump) for Unrecoverable Hard Lockups

If hard lockups continue to occur due to proprietary hardware controllers or obscure kernel bugs, configuring kdump captures a physical memory dump (vmcore) at the moment of panic for post-mortem debugging with crash:

# Install kdump tooling on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y kdump-tools crash linux-image-$(uname -r)-dbg

# Or on RHEL / AlmaLinux / Rocky Linux
sudo dnf install -y kexec-tools crash

# Verify kdump service status
sudo kdump-config status

6. Enterprise Bare-Metal Infrastructure Architecture in Pakistan

In shared cloud hosting and multi-tenant VPS environments, underlying hardware resources are dynamically overcommitted. When neighboring virtual machines exhaust CPU pipelines or saturate shared storage SAN fabrics, hypervisor CPU scheduling stalls and host PCIe queue freezes manifest inside tenant instances as inexplicable soft lockups and RCU freezes.

  • Dedicated Bare-Metal Compute: Running mission-critical ERP systems, high-frequency financial platforms, and transactional e-commerce stores on physical Dedicated Servers eliminates the hypervisor layer entirely. With 100% dedicated hardware cores, uncontended PCIe NVMe channels, and zero CPU steal time, kernel execution threads run unimpeded.
  • Local Pakistan Data Center Redundancy: For banking organizations, payment processors, and domestic enterprises, hosting on Dedicated Servers in Pakistan delivers sub-10ms domestic latency over direct Pakistan Internet Exchange (PKIX) fiber interconnects, robust dual-feed data center power, and dedicated enterprise hardware RAID controllers with battery-backed write cache (BBU) to completely prevent I/O-induced kernel lockups.
High-Reliability Bare-Metal Infrastructure

Eliminate Virtualization Lockups with Dedicated Servers in Pakistan

Say goodbye to hypervisor steal time, noisy-neighbor lockups, and storage I/O stalls. Deploy enterprise dedicated bare-metal servers featuring dedicated AMD EPYC and Intel Xeon processors, enterprise PCIe 4.0 NVMe storage, and unmetered domestic bandwidth in Karachi and Lahore.

Explore Pakistan Dedicated Servers → View Global Dedicated Servers
Sources & Official References
  1. Linux Kernel Organization. (2024). Lockup Detectors (Softlockup and Hardlockup Watchdogs) Documentation. The Linux Kernel Documentation Project. https://www.kernel.org/doc/html/latest/admin-guide/lockup-watchdogs.html
  2. Red Hat Customer Portal. (2023). What is a Soft Lockup and How to Troubleshoot it on RHEL. Red Hat Knowledgebase. https://access.redhat.com/solutions/19443
  3. SUSE Documentation. (2024). Understanding and Diagnosing Kernel Hard Lockups with NMI Watchdog. SUSE Knowledge Base. https://www.suse.com/support/kb/doc/?id=000018898
  4. McKenney, P. E. (2023). What is RCU, Fundamentally? RCU Grace Period Mechanics and Stall Detection. The Linux Kernel Documentation. https://www.kernel.org/doc/html/latest/RCU/whatisRCU.html
  5. Ubuntu Server Guide. (2024). Kernel Tuning and Sysctl Watchdog Configuration for Production Deployments. Canonical Documentation. https://ubuntu.com/server/docs

Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.

Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.