Troubleshooting Linux 'systemd-oomd Killed Process (cgroup memory pressure exceeds limit)': Complete PSI Runbook

Under modern Linux distributions—such as Ubuntu 22.04 LTS, Ubuntu 24.04 LTS, Debian 12 (Bookworm), Fedora 36+, and RHEL 9—production database administrators and DevOps engineers have increasingly encountered an aggressive, baffling failure mode: critical backend services abruptly vanish.

Without warning, an active MySQL or MariaDB instance terminates mid-transaction, an entire PHP-FPM worker pool is dropped during a traffic surge, or a Docker engine container crashes. When administrators immediately run diagnostic checks, the system state makes no intuitive sense:

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           64120       18410       32450         420       13260       44850
Swap:           8192         512        7680

The server boasts over 30 GB of completely free physical RAM, gigabytes of untouched swap space, and dmesg -T reveals zero invocations of the legacy kernel Out-of-Memory (OOM) killer. Yet, inspecting the systemd journal reveals the culprit:

systemd[1]: systemd-oomd.service: Memory pressure for slice user.slice exceeds limit, killing cgroup /user.slice/user-1000.slice/[email protected]/app.slice/...
systemd[1]: systemd-oomd.service: Killed /system.slice/mysql.service due to memory pressure (54.89% > 54.00% for >20s).
systemd[1]: mysql.service: A process of this unit has been killed by the OOM killer.
systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
systemd[1]: mysql.service: Failed with result 'oom-kill'.

The service was murdered not by physical exhaustion of hardware RAM, but by systemd-oomd: a proactive userland daemon operating on kernel Pressure Stall Information (PSI) metrics within the control group v2 (cgroup v2) subsystem.

This exhaustive engineering runbook unpacks the internal architecture of systemd-oomd, explains why default out-of-the-box configurations trigger catastrophic false-positive terminations on high-throughput database and web workloads, and provides a field-tested production override blueprint to permanently protect mission-critical services.

---

1. Internal Architecture: systemd-oomd vs. Legacy Kernel OOM Killer

To debug and eliminate unexpected service kills, system engineers must understand the architectural distinction between the Linux kernel’s internal OOM killer and userland pressure daemons.

The Problem with the Legacy Kernel OOM Killer

Historically, the Linux kernel Out-of-Memory killer (mm/oom_kill.c) remained completely dormant until the system reached absolute physical memory exhaustion. The kernel would only intervene when an allocation request in alloc_pages() failed, direct reclaim could not free any pages, and no unallocated pages remained in physical RAM or swap.

[ Kernel OOM Architecture (Legacy) ]
Physical RAM Exhaustion -> Direct Reclaim Fails -> High CPU Thrashing (kswapd) -> Complete System Lockup -> Kernel OOM Kills Process (Often Minutes Late)

In high-concurrency enterprise environments, this reactive mechanism caused catastrophic livelocks:

• Before the kernel OOM killer engaged, the kernel thread kswapd would enter direct page reclaim, spinning CPU cores at 100% trying to scan LRU lists and flush dirty pages to disk.
• Storage I/O queues saturated completely under thrashing.
• SSH connections dropped, metrics agents timed out, and the machine froze for minutes or hours in an unresponsive “zombie” state before the kernel finally selected and killed a process.

The Modern Paradigm: systemd-oomd and Pressure Stall Information (PSI)

Introduced to mainline Linux by Facebook/Meta engineers and integrated into systemd v247+, systemd-oomd shifts memory management from reactive survival to proactive prevention. Rather than waiting for total memory exhaustion, systemd-oomd continuously monitors resource contention metrics exported by the kernel via Pressure Stall Information (PSI) in /proc/pressure/memory and individual cgroup v2 directories.

[ systemd-oomd Architecture (cgroup v2 + PSI) ]
Continuous Monitored Loop (/proc/pressure/memory) 
  --> Tracks Stall Duration (some avg10 / full avg10)
  --> Detects cgroup Exceeding Pressure Limit (e.g., 54% over 10s or 20s)
  --> Identifies Culprit Control Group (system.slice, user.slice, container slice)
  --> Sends SIGKILL via cgroup.kill BEFORE Kernel Locks Up

Operating in user space, systemd-oomd leverages inotify and Linux kernel epoll triggers on /sys/fs/cgroup/.../memory.pressure. When memory stall times exceed configured limits, systemd-oomd acts decisively: it kills an entire target control group (cgroup) using the modern kernel interface cgroup.kill to terminate every related PID instantaneously.

Feature / Dimension	Legacy Kernel OOM Killer (mm/oom_kill.c)	Modern Userland Daemon (systemd-oomd)
Execution Space	Kernel space (alloc_pages slowpath)	User space daemon running as system service
Trigger Mechanism	Complete physical allocation failure (0 free pages)	Memory Pressure Stall Information (PSI thresholds) & Swap utilization
Kill Scope	Single process (task_struct via PID)	Entire control group (cgroup tree / slice)
Latency to Action	Very slow; often preceded by minutes of severe I/O thrashing	Fast and proactive (seconds after stall threshold breach)
Awareness of cgroup Limits	Rudimentary / Per-process heuristics (oom_score)	Native cgroup v2 hierarchies, slices, and resource boundaries
Customizability	Fixed sysctl parameters (vm.panic_on_oom, etc.)	Per-unit drop-in files (systemctl edit), global oomd.conf

---

2. The Unexpected Kill Catastrophe: Free RAM vs. Memory Pressure

The primary reason system administrators find systemd-oomd frustrating is the disconnect between free memory volume and memory pressure stalls.

Deconstructing Pressure Stall Information (PSI)

The Linux kernel quantifies resource starvation under /proc/pressure/memory. You can view real-time host-wide pressure metrics:

$ cat /proc/pressure/memory
some avg10=18.42 avg60=8.14 avg300=2.01 total=48192834
full avg10=4.12 avg60=1.85 avg300=0.45 total=12401928

PSI defines two distinct metrics:

1. some: The percentage of wall-clock time in which at least some non-idle tasks were delayed or stalled waiting for memory access (such as waiting for pages to be paged in from disk or waiting for page reclaim).
2. full: The percentage of wall-clock time in which all runnable tasks were simultaneously stalled waiting for memory. During full stalls, CPU processing capacity is completely wasted because no execution thread can proceed until memory I/O resolves.

The numbers avg10, avg60, and avg300 represent exponential moving averages over 10-second, 60-second, and 300-second windows.

Why systemd-oomd Kills Services with Ample Free RAM

In standard installations (notably Ubuntu and Fedora), systemd ships with default units that attach OOM pressure rules directly to entire system slices:

# Default system-level slice policy
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=54%

The trigger condition is deceptively simple: If some memory pressure exceeds 54% over a rolling 10-second window, systemd-oomd identifies the largest cgroup within that slice and terminates it with SIGKILL.

Consider what happens on a production host:

• A database server with 64 GB of physical RAM has 32 GB unallocated (free).
• A complex SQL query or sudden disk flush initiates heavy storage I/O.
• Because storage I/O or swap paging causes thread allocation delays, tasks spend 5.5 seconds out of a 10-second window stalled waiting on page cache locks or memory bus access.
• The kernel reports some avg10=55.20.
• Despite having 32 GB of pristine, unallocated RAM sitting untouched, systemd-oomd detects a breach of the 54% threshold and forcefully kills MySQL or MariaDB.

---

3. Four Core Environments Triggering systemd-oomd Service Terminations

In production server management, systemd-oomd kills clustered primarily around four specific application architectures and operational patterns.

+---------------------------------------------------------------------------------------+
|                       COMMON SYSTEMD-OOMD KILL TRIGGERS                                |
+---------------------------------------------------------------------------------------+
|  1. High-Concurrency Database Bursts                                                  |
|     InnoDB temp tables, buffer pool flushes, and massive table scans trigger PSI.     |
+---------------------------------------------------------------------------------------+
|  2. Web Server & PHP-FPM Traffic Spikes                                               |
|     Burst of child worker processes exceeding cgroup memory pressure limits.          |
+---------------------------------------------------------------------------------------+
|  3. Multi-Tenant CI/CD & Build Runaways                                               |
|     Docker builds, webpack/npm compilations, or Go/Rust compiler concurrency spikes.  |
+---------------------------------------------------------------------------------------+
|  4. System Slice Swap Starvation                                                      |
|     Swapless hosts or tiny swap partitions crossing SwapUsedLimit=90% threshold.      |
+---------------------------------------------------------------------------------------+

Environment 1: High-Concurrency Database Bursts (MySQL / MariaDB)

Relational databases like MySQL and MariaDB manage massive internal memory structures (such as innodb_buffer_pool_size). During heavy transactional volume—such as flash sales on large e-commerce platforms or large data imports:

• Queries that cannot execute in-memory allocate disk-backed temporary tables (tmp_table_size, max_heap_table_size).
• If dirty buffer flushing backs up, thread allocation calls stall waiting for page cleanups.
• Even if total physical RAM is never exhausted, the synchronous delay across multiple concurrent client threads inflates /sys/fs/cgroup/system.slice/mysql.service/memory.pressure past the 54% threshold.
• systemd-oomd selects mysql.service because its RSS is the largest footprint in system.slice, abruptly dropping client connections and causing data corruption risks.

Environment 2: WordPress & PHP-FPM Dynamic Spikes

In shared hosting, cPanel environments, or multi-tenant web servers:

• Traffic surges cause php-fpm to spawn dozens of child processes up to pm.max_children.
• Each PHP worker processes dynamic WordPress requests, rendering complex themes, loading object cache records, and querying databases.
• The simultaneous allocation of memory across 50+ worker threads creates an instantaneous burst of memory pressure.
• systemd-oomd flags php8.2-fpm.service as exceeding the slice pressure limit, terminating the master daemon and all child workers, resulting in immediate HTTP 502 Bad Gateway errors across all hosted sites.

Environment 3: Multi-Tenant CI/CD & Container Build Runaways

On hosts running Docker, Podman, or Kubernetes alongside system daemons:

• A developer runs an intense build: docker build -t app ., invoking npm run build, cargo build --release, or multi-threaded C++/Go compiles.
• Compilers aggressively spawn parallel jobs (make -j$(nproc)), consuming all available memory bandwidth and triggering brief, severe page allocation stalls.
• If the build executes within a cgroup managed under system.slice or user.slice, systemd-oomd fires. Rather than simply throttling or failing the build process, it may select and terminate a co-located production background service that happened to occupy the highest memory footprint at that instant.

Environment 4: System Slice Swap Starvation

By default, systemd-oomd monitors swap utilization in addition to PSI memory metrics. In /etc/systemd/oomd.conf, the default directive is:

SwapUsedLimit=90%

On modern cloud virtual machines configured without swap space (or with an undersized 512 MB swapfile):

• Modest background activity fills the tiny swap space to 91%.
• systemd-oomd immediately concludes the system is in an unrecoverable swap emergency.
• It initiates aggressive cgroup terminations host-wide, indiscriminately killing services to restore swap headroom below 90%, even when physical RAM utilization is below 30%.

---

4. Real-Time Diagnostic Procedures & Telemetry

When a production daemon terminates unexpectedly, follow this forensic runbook to confirm whether systemd-oomd was the executioner and analyze the exact metrics that triggered the kill.

Step 1: Inspect systemd-oomd Termination Logs

Check the journal logs of the systemd-oomd daemon specifically:

journalctl -u systemd-oomd.service --no-pager -n 50

If systemd-oomd triggered the kill, you will see output formatted like this:

Oct 28 14:22:04 web-prod-01 systemd-oomd[741]: Memory pressure for slice system.slice exceeds limit (58.41% > 54.00% for >20s), killing cgroup /system.slice/mysql.service
Oct 28 14:22:04 web-prod-01 systemd-oomd[741]: Killed /system.slice/mysql.service due to memory pressure

Cross-reference this with unit-level kill records:

journalctl -xeu mysql.service --no-pager | grep -i oom

Output:

systemd[1]: mysql.service: A process of this unit has been killed by the OOM killer.

Step 2: Inspect Live Status with oomctl

Systemd provides a dedicated command-line utility, oomctl, to inspect the current state of monitored control groups, active pressure limits, and recent kills:

oomctl

Example production output:

Dry Run: no
Swap Target: /
  Swap Total: 8.0G
  Swap Used: 412.0M (5.0%)
Memory Pressure Limit: 54.00%
Default Memory Pressure Duration: 20s
System Context:
  Path: /
    Memory Pressure: some avg10=2.15 avg60=1.40 avg300=0.85
    Swap Used: 412.0M (5.0%)
Monitored cgroups:
  Path: /system.slice
    Memory Pressure Limit: 54.00%
    Pressure: some avg10=1.12 avg60=0.98 avg300=0.45
    Current Duration: 0s
  Path: /system.slice/mysql.service
    Memory Pressure Limit: auto
    Pressure: some avg10=0.00 avg60=0.00 avg300=0.00
  Path: /user.slice
    Memory Pressure Limit: 54.00%
    Pressure: some avg10=0.00 avg60=0.00 avg300=0.00

Pay special attention to Memory Pressure Limit and Current Duration. If Current Duration is ticking upwards toward the duration threshold (e.g., 20s), the cgroup is currently under evaluation for termination.

Step 3: Monitor Live PSI Metrics

To observe host-level memory stall progression under load:

cat /proc/pressure/memory

To monitor real-time pressure continuously with sub-second fidelity, run:

watch -n 1 "cat /proc/pressure/memory; echo '---'; cat /proc/pressure/io; echo '---'; cat /proc/pressure/cpu"

To view the pressure metric inside an individual service’s cgroup:

cat /sys/fs/cgroup/system.slice/mysql.service/memory.pressure

Output:

some avg10=0.00 avg60=0.00 avg300=0.00 total=14209
full avg10=0.00 avg60=0.00 avg300=0.00 total=3102

Step 4: Real-Time cgroup Monitoring with systemd-cgtop

To identify which services are consuming the greatest memory bandwidth and generating the highest I/O pressure:

systemd-cgtop -m

This orders all active cgroups dynamically by memory consumption, showing allocation rates and task counts in real time.

Step 5: Check Process OOM Scores and Adjustments

Even when systemd-oomd acts via PSI, it uses internal heuristics and kernel oom_score parameters to choose which process tree to kill first within an offending slice.

Inspect a process’s badness score:

PID=$(pgrep -f mysqld | head -n 1)
cat /proc/$PID/oom_score
cat /proc/$PID/oom_score_adj

• oom_score: An integer between 0 and 1000 calculated by the kernel reflecting how much memory the process consumes relative to system limits. Higher values indicate higher priority for termination.
• oom_score_adj: An adjustment factor between -1000 (completely exempt from OOM kill) to +1000 (first in line to be killed).

---

5. Actionable Configuration Runbook & System Overrides

To ensure your critical database, web, and container services are never terminated by systemd-oomd, apply the following production configuration overrides.

Strategy 1: Protect Critical Daemons via Systemd Drop-in Overrides

You can configure per-unit protection directly within the service’s systemd unit configuration using systemctl edit.

To protect MySQL or MariaDB:

sudo systemctl edit mysql.service

Paste the following configuration into the override file:

[Service]
# Prevent systemd-oomd from monitoring this unit for kill action
ManagedOOMMemoryPressure=auto
ManagedOOMMemoryPressureLimit=85%
ManagedOOMPreference=avoid

# Protect process from kernel-level OOM killer as well
OOMScoreAdjust=-900

To protect PHP-FPM (for instance, php8.2-fpm):

sudo systemctl edit php8.2-fpm.service

[Service]
ManagedOOMPreference=avoid
OOMScoreAdjust=-800

To protect Docker or Containerd:

sudo systemctl edit docker.service

[Service]
ManagedOOMPreference=avoid
OOMScoreAdjust=-900

Explanation of Directives:

• ManagedOOMPreference=avoid: Instructs systemd-oomd that when the parent slice (e.g., system.slice) experiences excessive pressure, this specific cgroup should be avoided if any other candidate cgroups exist.
• ManagedOOMPreference=omit: Completely exempts the cgroup from being targeted under any circumstances when slice-level pressure thresholds are breached.
• ManagedOOMMemoryPressure=auto: Inherits memory pressure management from parent slices rather than enforcing strict kill rules on this specific service.
• ManagedOOMMemoryPressureLimit=85%: If pressure monitoring is active, relaxes the threshold from the aggressive 54% default up to 85%.
• OOMScoreAdjust=-900: Drastically reduces the process’s kernel OOM badness score. A score of -1000 grants absolute immunity from the kernel OOM killer.

After modifying overrides, reload the systemd manager configuration and restart the unit if necessary:

sudo systemctl daemon-reload
sudo systemctl restart mysql.service

Verify that the changes are active:

oomctl

Strategy 2: Global systemd-oomd Configuration in /etc/systemd/oomd.conf

Instead of (or in addition to) per-service overrides, you can tune the global sensitivity of systemd-oomd across the entire operating system.

Edit /etc/systemd/oomd.conf:

sudo nano /etc/systemd/oomd.conf

Configure production-grade thresholds:

[OOM]
# Relax default pressure limit from 54% to 80%
DefaultMemoryPressureLimit=80%

# Extend stall duration required before triggering a kill from 20s to 30s
DefaultMemoryPressureDurationSec=30s

# Raise swap limit to prevent premature kills on small swap setups
SwapUsedLimit=95%

Restart the systemd-oomd daemon to apply the new global policies:

sudo systemctl restart systemd-oomd.service

Verify the active global parameters:

oomctl

Strategy 3: Configure Swap to Prevent Swap Starvation Kills

If your server runs with little or no swap, SwapUsedLimit=90% will routinely trigger false-positive kills. Ensure your host has an adequate, NVMe-backed swapfile:

# Create a 4 GB swapfile if missing
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Add to /etc/fstab for persistence
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Tune kernel swappiness to a conservative production value
sudo sysctl -w vm.swappiness=10
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.d/99-sysctl.conf

Strategy 4: Disabling or Masking systemd-oomd on Dedicated Servers

On single-purpose, high-throughput dedicated database servers (such as bare-metal machines running dedicated MySQL, PostgreSQL, or Redis clusters), userland memory pressure killers often introduce more operational risk than stability.

On a dedicated database server, the database is the operating system’s primary payload. Having a userland daemon kill MySQL because it generated temporary memory pressure during a batch index build or complex query is completely unacceptable.

To safely disable systemd-oomd entirely:

# Stop and disable the service
sudo systemctl stop systemd-oomd.service
sudo systemctl disable systemd-oomd.service

# Mask the service to prevent activation by sockets or dependencies
sudo systemctl mask systemd-oomd.service

Verify that systemd-oomd is masked:

systemctl status systemd-oomd.service

Output:

Loaded: masked (reason: Unit systemd-oomd.service is masked.)
Active: inactive (dead)

With systemd-oomd disabled, memory management falls back to the traditional kernel memory reclaim algorithms (kswapd, direct reclaim) and the kernel OOM killer (mm/oom_kill.c), which will only engage if physical RAM and swap are genuinely 100% depleted.

---

6. Infrastructure Architecture: Virtual Hypervisor Jitter vs. Dedicated Bare Metal

While tuning cgroup parameters and overriding systemd limits resolves software-level triggers, high-concurrency production platforms frequently face an underlying physical hardware challenge: hypervisor-induced PSI inflation.

The Hypervisor Noise Problem

In virtualized cloud environments (public cloud VPS, shared hypervisors, oversubscribed cloud slices):

1. Memory Ballooning: Hypervisors use balloon drivers (virtio_balloon) to dynamically reclaim memory from guest VMs to satisfy demands of neighboring tenants on the same physical host.
2. CPU Scheduling Steal Time: When the underlying physical CPU is oversubscribed, guest vCPU threads pause during memory allocation routines. The Linux kernel’s PSI subsystem measures this delay as memory stall time (some avg10), even though the guest OS has plenty of assigned RAM.
3. Storage I/O Latency Spikes: Shared SAN, Ceph, or network-attached block storage frequently suffers from latency spikes during multi-tenant I/O bursts. Dirty page writebacks freeze, page cache reclaims stall, and PSI memory pressure exceeds the 54% threshold instantly.

+-------------------------------------------------------------------------+
|                  VIRTUALIZED MULTI-TENANT HYPERVISOR                    |
|                                                                         |
|  [ Guest Linux OS ]       <-- Virtualized vCPUs & Dynamic Memory        |
|           |                                                             |
|           v                                                             |
|  [ Hypervisor Balloon ]   <-- Steals Memory; Causes Artificial PSI Spikes|
|           |                                                             |
|           v                                                             |
|  [ Shared Storage SAN ]   <-- I/O Latency Freezes Dirty Page Writebacks  |
|           |                                                             |
|           v                                                             |
|  [ Shared Physical RAM ]  <-- Unpredictable Memory Pressure & Stalls     |
+-------------------------------------------------------------------------+

                                   VS

+-------------------------------------------------------------------------+
|                 NEXTGEN ENTERPRISE DEDICATED HARDWARE                   |
|                                                                         |
|  [ Bare-Metal Kernel ]    <-- Direct Control of Physical Hardware       |
|           |                                                             |
|           v                                                             |
|  [ Dedicated DDR5 ECC ]   <-- 100% Deterministic Low-Latency Bus        |
|           |                                                             |
|           v                                                             |
|  [ Direct PCIe NVMe ]     <-- Sub-millisecond Flush & Page I/O          |
|           |                                                             |
|           v                                                             |
|  [ Zero Ballooning ]      <-- PSI Reflects True Hardware State Only     |
+-------------------------------------------------------------------------+

The Bare-Metal Resolution

For mission-critical transactional platforms—including high-volume WooCommerce stores, enterprise Redis cache layers, PostgreSQL data warehouses, and high-concurrency container clusters—eliminating virtualization jitter is paramount.

Deploying on bare-metal Dedicated Servers gives your workload dedicated physical multi-channel DDR5 ECC RAM, deterministic NUMA node architecture, and unthrottled direct PCIe Gen4/Gen5 NVMe storage arrays. Without a hypervisor stealing CPU cycles or ballooning memory behind the scenes, memory pressure stalls remain near zero under normal operation.

For businesses targeting low latency, regulatory data sovereignty, and regional connectivity in South Asia, deploying on Dedicated Servers in Pakistan provides single-hop routing, local BGP peering, and enterprise-grade dedicated hardware that insulates your mission-critical applications from false-positive memory terminations.

---

7. Production Verification & Diagnostic Cheat Sheet

Use this reference table to quickly audit and troubleshoot systemd-oomd on any Linux host:

Diagnostic Goal	Production Command
Inspect OOM Kill Events	journalctl -u systemd-oomd -n 50 --no-pager
View oomctl Telemetry	oomctl
Query Host Memory Pressure	cat /proc/pressure/memory
Query Host I/O Pressure	cat /proc/pressure/io
Query Service-Level Pressure	cat /sys/fs/cgroup/system.slice/<service>.service/memory.pressure
Live cgroup Memory Sorting	systemd-cgtop -m
Check Process OOM Score	cat /proc/<PID>/oom_score
Check Process OOM Adjust	cat /proc/<PID>/oom_score_adj
Edit Service Override	sudo systemctl edit <service>.service
Reload Systemd Daemon	sudo systemctl daemon-reload
Restart systemd-oomd	sudo systemctl restart systemd-oomd
Mask systemd-oomd	sudo systemctl mask --now systemd-oomd

---