Production Linux servers powering high-concurrency relational databases, virtualization hypervisors, and distributed storage nodes occasionally suffer from sudden system unresponsiveness, erratic latency spikes, or kernel panics. During post-incident analysis, system administrators frequently find alarming traces inside the kernel ring buffer (dmesg), such as watchdog: BUG: soft lockup - CPU#4 stuck for 22s! or rcu: INFO: rcu_sched self-detected stall on CPU.[1]
These kernel alarms indicate that a logical CPU core is stuck executing kernel-space code without yielding execution to the OS scheduler or processing hardware interrupts within the configured watchdog threshold. If left unmitigated, watchdog events trigger unpredictable cascading timeouts across TCP sockets, cluster heartbeats (Corosync/Pacemaker), and active database transactions. This runbook provides production-tested diagnostic commands, architectural explanations of lockup mechanisms, crash log dissection, and kernel parameter tuning to eliminate soft and hard lockups across bare-metal and enterprise virtualized infrastructure.
1. Architectural Anatomy of Linux Kernel Watchdogs
The Linux kernel incorporates three distinct internal watchdog mechanisms designed to monitor CPU responsiveness, prevent deadlock conditions, and force automated recovery when cores become completely wedged:
| Watchdog Subsystem | Detection Method | Interrupt State | Default Trigger Interval | Kernel Panic Control Parameter |
|---|---|---|---|---|
| Soft Lockup Watchdog | Per-CPU watchdog/N kernel thread (SCHED_FIFO) |
Local IRQs Enabled | $2 \times \text{watchdog_thresh}$ (20s) | kernel.softlockup_panic |
| Hard Lockup Watchdog | Non-Maskable Interrupts (NMI) via APIC / perf counters |
Local IRQs Disabled | $1 \times \text{watchdog_thresh}$ (10s) | kernel.hardlockup_panic |
| RCU Stall Detector | Periodic RCU GP timer checking quiescent states | Either Enabled or Disabled | rcu_cpu_stall_timeout (21s) |
kernel.panic_on_rcu_stall |
Soft Lockup Mechanism
A soft lockup occurs when a task executes inside kernel mode (such as inside a system call, memory allocation routine, or driver handler) for longer than twice the kernel.watchdog_thresh interval without yielding or calling schedule().[2]
Because local CPU interrupts remain enabled, timer interrupts continue to fire. A high-priority real-time kernel thread, watchdog/N (configured with scheduling policy SCHED_FIFO and priority 99), is woken periodically by a timer interrupt to touch a per-CPU timestamp (watchdog_touch_ts). A secondary high-resolution timer (hrtimer) monitors this timestamp. If the timestamp is older than $2 \times \text{watchdog_thresh}$ seconds, the kernel detects that the watchdog/N thread has been starved from execution and logs watchdog: BUG: soft lockup.
Hard Lockup Mechanism
A hard lockup occurs when a CPU is looping inside kernel space with local hardware interrupts disabled (e.g., via local_irq_disable() or inside a nested spin_lock_irqsave() critical section). Because local interrupts are masked, standard timer interrupts cannot fire, rendering the soft lockup thread useless.[3]
To catch hard lockups, the Linux kernel leverages the Local Advanced Programmable Interrupt Controller (APIC) and hardware performance monitoring units (PMU) via the perf subsystem to generate periodic Non-Maskable Interrupts (NMIs). Since NMIs cannot be masked by standard CPU interrupt flags, the NMI handler interrupts the stuck core, evaluates the cycle counters, and detects that the timer interrupt handler has stopped incrementing its tick counter (hrtimer_interrupts). When this occurs, the NMI watchdog emits watchdog: Watchdog detected hard LOCKUP on cpu.
RCU Grace Period Stalls
Read-Copy Update (RCU) is a scalable synchronization mechanism ubiquitous in the Linux kernel networking and VFS layers. Readers traverse data structures without acquiring expensive mutexes by disabling preemption inside rcu_read_lock() and rcu_read_unlock() blocks. Writers modify pointers atomically and defer the freeing of old memory until all existing readers have passed through a “quiescent state” (a context switch or CPU idle transition), completing an RCU grace period.[4]
If a CPU or task remains inside an RCU read-side critical section or fails to yield for longer than rcu_cpu_stall_timeout seconds, memory reclamation halts across the kernel, and the RCU stall detector logs rcu: INFO: rcu_sched self-detected stall on CPU.
2. Decrypting Kernel Stack Traces in dmesg
When a watchdog fires, the kernel dumps a detailed diagnostic report to dmesg and /var/log/messages. Analyzing the instruction pointer, call trace, and task context reveals the precise operational bottleneck.
Case 1: Soft Lockup with Memory Compaction Contention
[14285.104821] watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [mysqld:32145]
[14285.104828] Modules linked in: xfs ext4 dm_mod kvm_intel kvm irqbypass crct10dif_pclmul
[14285.104845] CPU: 4 PID: 32145 Comm: mysqld Kdump: loaded Tainted: G L 6.6.0-14-generic
[14285.104850] RIP: 0010:native_queued_spin_lock_slowpath+0x82/0x2a0
[14285.104855] Code: 48 89 f0 48 c1 e8 10 66 31 c0 f3 90 8b 07 66 85 c0 75 f7 8b 07
[14285.104860] RSP: 0018:ffffa8b48879b9d8 EFLAGS: 00000206
[14285.104863] RAX: 0000000000000101 RBX: ffff9a1c7c2b4e00 RCX: 0000000000000000
[14285.104866] RDX: 0000000000000000 RSI: 0000000000000101 RDI: ffff9a1c7c2b4e18
[14285.104870] Call Trace:
[14285.104872] <TASK>
[14285.104875] _raw_spin_lock+0x24/0x30
[14285.104878] compact_zone+0x4c2/0x890
[14285.104882] try_to_compact_pages+0x138/0x2c0
[14285.104886] __alloc_pages_direct_compact+0x85/0x1e0
[14285.104890] __alloc_pages_slowpath.constprop.0+0x412/0xba0
[14285.104894] __alloc_pages+0x21b/0x2a0
[14285.104898] alloc_pages_vma+0x8a/0x1f0
[14285.104902] do_huge_pmd_anonymous_page+0x14f/0x7b0
[14285.104906] __handle_mm_fault+0x8b2/0xe20
[14285.104910] handle_mm_fault+0xc5/0x280
[14285.104914] do_user_addr_fault+0x21e/0x650
[14285.104918] exc_page_fault+0x7b/0x170
[14285.104922] asm_exc_page_fault+0x27/0x30
[14285.104926] </TASK>
Diagnostic Interpretation:
- Triggering Process:
mysqld(PID 32145) running on CPU 4. - Fault Location (
RIP):native_queued_spin_lock_slowpath. The process is spinning on a kernel lock. - Call Trace Sequence: An anonymous page fault triggered
do_huge_pmd_anonymous_pageattempting to allocate Transparent Huge Pages (THP). Because memory is fragmented, the kernel invokedtry_to_compact_pagesand enteredcompact_zone. Multiple CPU cores attempted to acquire zone locks simultaneously, inducing extreme spinlock contention and starving the CPU for 22 seconds.
Case 2: Hard Lockup in Hardware Interrupt Handler
[89214.301211] watchdog: Watchdog detected hard LOCKUP on cpu 2
[89214.301215] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.6.0-14-generic
[89214.301218] RIP: 0010:megasas_complete_cmd_dpc+0x15a/0x320 [megaraid_sas]
[89214.301225] Call Trace:
[89214.301227] <NMI>
[89214.301230] watchdog_hardlockup_check+0x88/0x110
[89214.301234] perf_event_nmi_handler+0x28/0x50
[89214.301238] nmi_handle+0x62/0x160
[89214.301242] default_do_nmi+0x54/0x120
[89214.301246] exc_nmi+0xb8/0xe0
[89214.301249] asm_exc_nmi+0x31/0x40
[89214.301253] <<INTERRUPT>>
[89214.301256] _raw_spin_lock_irqsave+0x32/0x40
[89214.301260] megasas_complete_cmd_dpc+0x78/0x320 [megaraid_sas]
[89214.301264] tasklet_action_common.constprop.0+0xad/0x120
[89214.301268] handle_softirqs+0xdf/0x2c0
[89214.301272] irq_exit_rcu+0x89/0xb0
Diagnostic Interpretation:
- Triggering Component: SAS hardware RAID driver (
megaraid_sas) servicing pending I/O completion frames. - Failure Point: CPU 2 acquired a lock with
_raw_spin_lock_irqsave, disabling interrupts. The driver fell into an unrecoverable busy-loop waiting for a frozen controller register response, preventing interrupts from resuming for longer than 10 seconds. The hardware NMI watchdog fired and caught the lockup.
3. Root Cause Taxonomy of Production Kernel Lockups
Understanding the underlying operational conditions that lead to soft and hard lockups prevents misdirected troubleshooting:
1. Memory Compaction and Direct Reclaim Exhaustion
When physical memory allocation headroom is exhausted and memory is heavily fragmented, kernel tasks requiring high-order allocations (such as Transparent Huge Pages or jumbo network frames) enter direct reclaim and zone compaction. If many worker threads simultaneously loop in compact_zone() acquiring zone->lock, the system enters lock starvation, triggering soft lockup warnings.
2. Hypervisor vCPU Scheduling Inversion (Steal Time)
In cloud instances and virtualized VPS environments with overcommitted hypervisor hosts, the host scheduler may preempt a virtual CPU (vCPU) while that vCPU is holding an in-kernel spinlock inside the guest operating system. When other guest vCPUs attempt to acquire the same spinlock, they spin endlessly waiting for the descheduled vCPU to run again. If hypervisor steal time exceeds 20 seconds, the guest kernel logs a soft lockup.
3. Disk I/O Stalls and Dirty Page Flush Locks
When write-heavy workloads flood the page cache faster than underlying NVMe or SAS controllers can commit data to non-volatile flash, the kernel throttles dirty page generation. Tasks attempting disk writes get blocked on inode locks (ext4_writepages, xfs_vm_writepage) or dm-crypt encryption queues, accumulating long lockup intervals.
4. Broken Driver Loops and Hardware Controller Freezes
Faulty firmware in network adapters (e.g., ancient Realtek or Mellanox driver firmwares) or failing PCIe devices that stop acknowledging interrupts force kernel driver interrupt handlers into infinite polling loops with interrupts disabled, resulting in hard lockups.
4. Live Diagnostic and Forensic Runbook
When diagnosing an active or past lockup incident, follow this structured diagnostic checklist:
Step 1: Scan the Kernel Log for Watchdog Signatures
Execute a timestamped search across the kernel ring buffer and syslog archives:
# Search current kernel ring buffer for all watchdog, RCU, and lockup warnings
sudo dmesg -T | grep -E -i "soft lockup|hard lockup|rcu_sched|watchdog|blocked for more than"
# Search historical persistent journal logs
sudo journalctl -k --since "24 hours ago" | grep -E -i "soft lockup|hard lockup|rcu_sched|hung_task"
Step 2: Inspect Current Kernel Watchdog Configuration
Verify the active watchdog parameters and status across the running system:
# Query all active watchdog parameters via sysctl
sudo sysctl -a | grep -E "kernel\.(watchdog|softlockup|hardlockup|nmi_watchdog|panic_on)"
Expected standard production baseline:
kernel.hardlockup_all_cpu_backtrace = 0
kernel.hardlockup_panic = 0
kernel.nmi_watchdog = 1
kernel.panic_on_rcu_stall = 0
kernel.soft_watchdog = 1
kernel.softlockup_all_cpu_backtrace = 0
kernel.softlockup_panic = 0
kernel.watchdog = 1
kernel.watchdog_thresh = 10
Step 3: Monitor CPU Interrupts and Steal Time
Determine whether CPU cores are suffering from hardware IRQ imbalances or hypervisor preemption:
# Monitor per-CPU interrupts every 2 seconds
watch -n 2 -d "cat /proc/interrupts | head -n 30"
# Check CPU steal time using mpstat (sysstat package)
mpstat -P ALL 2 5
Pay special attention to the %steal column. If %steal spikes above 10% on virtualized instances, the root cause is external host hypervisor contention rather than guest software failure.
Step 4: Trace Kernel Lock Contention with perf
To pinpoint which kernel functions are consuming cycles spinning on locks during heavy load:
# Record kernel call traces for lock contention analysis
sudo perf top --sort comm,dso,symbol -e cycles:k
Look for symbols such as native_queued_spin_lock_slowpath, _raw_spin_lock, or mutex_spin_on_owner. If these dominate the top entries, locate the calling subsystem using the interactive perf report call-graph trace.
5. Kernel Tuning & Production Hardening Runbook
Implement the following production-tested configuration adjustments to mitigate soft lockups, prevent spurious panics, and optimize memory allocation performance.
1. Tune Watchdog Threshold for Heavy Server Workloads
On high-throughput bare-metal servers, large multi-socket NUMA database nodes, or virtualized hosts where transient load bursts can cause momentary scheduling delays, raising watchdog_thresh prevents false-positive panics:[5]
# Increase the watchdog threshold from 10 to 20 seconds
# Soft lockups will trigger after 40 seconds instead of 20 seconds
sudo sysctl -w kernel.watchdog_thresh=20
# Persist configuration across reboots
sudo bash -c 'cat <<EOF > /etc/sysctl.d/99-kernel-watchdog.conf
# Extend kernel watchdog threshold to absorb transient I/O spikes
kernel.watchdog_thresh = 20
# Avoid automated kernel panic on non-fatal soft lockups
kernel.softlockup_panic = 0
# Prevent panic on RCU stalls
kernel.panic_on_rcu_stall = 0
# Enable all CPU backtraces to capture full system state during a lockup
kernel.softlockup_all_cpu_backtrace = 1
kernel.hardlockup_all_cpu_backtrace = 1
EOF'
# Reload sysctl configurations
sudo sysctl --system
2. Mitigate Transparent Huge Pages (THP) Compaction Freezes
Memory compaction stalls during THP allocations represent one of the most common causes of mysqld, postgres, and java soft lockups. Reconfigure THP defragmentation from aggressive synchronous mode (always) to asynchronous background compaction (madvise or defer):
# Check current THP defragmentation mode
cat /sys/kernel/mm/transparent_hugepage/defrag
# Reconfigure THP defragmentation to avoid synchronous stall locks
echo "madvise" | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
# For database servers (MySQL, MariaDB, PostgreSQL), disable THP completely:
echo "never" | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo "never" | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
# Persist via systemd or GRUB boot options (/etc/default/grub)
# Add "transparent_hugepage=never" to GRUB_CMDLINE_LINUX_DEFAULT
3. Tune Virtual Memory Dirty Ratios to Prevent Storage Flush Hangs
When write bursts overwhelm storage controllers, the kernel blocks write system calls until dirty pages are flushed to disk. Adjusting dirty ratios forces the kernel to write data continuously in smaller background batches rather than executing huge, blocking synchronous flushes:
sudo bash -c 'cat <<EOF > /etc/sysctl.d/99-vm-dirty-tuning.conf
# Start background flushing when dirty memory hits 5% of RAM
vm.dirty_background_ratio = 5
# Block write operations only when dirty memory reaches 15% of RAM
vm.dirty_ratio = 15
# Reduce the age threshold for dirty pages to be written out (in centiseconds)
vm.dirty_expire_centisecs = 1500
vm.dirty_writeback_centisecs = 500
EOF'
sudo sysctl --system
4. Enable Automated Crash Dumps (kdump) for Unrecoverable Hard Lockups
If hard lockups continue to occur due to proprietary hardware controllers or obscure kernel bugs, configuring kdump captures a physical memory dump (vmcore) at the moment of panic for post-mortem debugging with crash:
# Install kdump tooling on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y kdump-tools crash linux-image-$(uname -r)-dbg
# Or on RHEL / AlmaLinux / Rocky Linux
sudo dnf install -y kexec-tools crash
# Verify kdump service status
sudo kdump-config status
6. Enterprise Bare-Metal Infrastructure Architecture in Pakistan
In shared cloud hosting and multi-tenant VPS environments, underlying hardware resources are dynamically overcommitted. When neighboring virtual machines exhaust CPU pipelines or saturate shared storage SAN fabrics, hypervisor CPU scheduling stalls and host PCIe queue freezes manifest inside tenant instances as inexplicable soft lockups and RCU freezes.
- Dedicated Bare-Metal Compute: Running mission-critical ERP systems, high-frequency financial platforms, and transactional e-commerce stores on physical Dedicated Servers eliminates the hypervisor layer entirely. With 100% dedicated hardware cores, uncontended PCIe NVMe channels, and zero CPU steal time, kernel execution threads run unimpeded.
- Local Pakistan Data Center Redundancy: For banking organizations, payment processors, and domestic enterprises, hosting on Dedicated Servers in Pakistan delivers sub-10ms domestic latency over direct Pakistan Internet Exchange (PKIX) fiber interconnects, robust dual-feed data center power, and dedicated enterprise hardware RAID controllers with battery-backed write cache (BBU) to completely prevent I/O-induced kernel lockups.
Eliminate Virtualization Lockups with Dedicated Servers in Pakistan
Say goodbye to hypervisor steal time, noisy-neighbor lockups, and storage I/O stalls. Deploy enterprise dedicated bare-metal servers featuring dedicated AMD EPYC and Intel Xeon processors, enterprise PCIe 4.0 NVMe storage, and unmetered domestic bandwidth in Karachi and Lahore.
Sources & Official References
- Linux Kernel Organization. (2024). Lockup Detectors (Softlockup and Hardlockup Watchdogs) Documentation. The Linux Kernel Documentation Project. https://www.kernel.org/doc/html/latest/admin-guide/lockup-watchdogs.html
- Red Hat Customer Portal. (2023). What is a Soft Lockup and How to Troubleshoot it on RHEL. Red Hat Knowledgebase. https://access.redhat.com/solutions/19443
- SUSE Documentation. (2024). Understanding and Diagnosing Kernel Hard Lockups with NMI Watchdog. SUSE Knowledge Base. https://www.suse.com/support/kb/doc/?id=000018898
- McKenney, P. E. (2023). What is RCU, Fundamentally? RCU Grace Period Mechanics and Stall Detection. The Linux Kernel Documentation. https://www.kernel.org/doc/html/latest/RCU/whatisRCU.html
- Ubuntu Server Guide. (2024). Kernel Tuning and Sysctl Watchdog Configuration for Production Deployments. Canonical Documentation. https://ubuntu.com/server/docs
Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.
