Diagnosing cgroup v2 Memory Pressure Stalls (PSI) & OOM Kills in Docker/LXC

Modern Linux distributions have almost universally adopted cgroup v2, bringing significant changes to how containerized workloads (like Docker, Podman, and LXC) manage system resources. While cgroup v2 offers superior resource isolation, it fundamentally alters how memory limits are enforced, often leading to seemingly inexplicable Out-Of-Memory (OOM) kills and severe system sluggishness before a crash.

A critical component of this new architecture is PSI (Pressure Stall Information), a kernel feature that measures the exact percentage of time tasks are blocked waiting for hardware resources—primarily memory.

If your Docker containers or LXC instances are spontaneously dying, or if you’re experiencing massive latency spikes before OOM events, you are likely hitting cgroup v2 memory pressure stalls. Here is a definitive, deep-knowledge guide to diagnosing and resolving these issues.

1. The cgroup v2 Memory Throttle Hierarchy

Unlike cgroup v1, which primarily relied on hard limits (memory.limit_in_bytes), cgroup v2 employs a multi-tiered approach to memory management designed to proactively slow down greedy processes before invoking the destructive OOM killer.

The three critical memory thresholds in a cgroup v2 environment are:

• memory.low: Best-effort protection. If a container’s memory usage is below this threshold, the kernel will try to avoid reclaiming memory from it during system-wide memory pressure.
• memory.high: The soft limit (throttle point). If a container breaches this limit, the kernel does not kill the process. Instead, it heavily throttles the cgroup by forcing processes into sleep states (stalls) while the kernel aggressively reclaims memory. This is what causes severe performance degradation and high PSI readings.
• memory.max: The hard limit. If memory usage reaches this point and the kernel cannot reclaim enough pages, the OOM killer is triggered immediately.

2. Deciphering PSI (Pressure Stall Information)

Before a container hits memory.max and gets OOM-killed, it almost certainly hits memory.high and experiences memory pressure stalls. You can monitor this in real-time via PSI.

Check the host’s global memory pressure:

cat /proc/pressure/memory

Output:

some avg10=15.23 avg60=5.41 avg300=1.12 total=134560
full avg10=4.10 avg60=0.82 avg300=0.20 total=42100

• some: The percentage of time in the last 10, 60, or 300 seconds that at least one task in the system (or cgroup) was delayed due to lack of memory.
• full: The percentage of time that all non-idle tasks were stalled simultaneously. A high full average means the system is thrashing (constantly reclaiming memory or swapping) and doing zero productive work.

You can inspect PSI for a specific Docker container by finding its cgroup path:

# Find the container's cgroup path
CONTAINER_ID=$(docker ps -qf "name=my-app-db")
CGROUP_PATH=$(find /sys/fs/cgroup -name "*${CONTAINER_ID}*")

# View the container's memory pressure
cat ${CGROUP_PATH}/memory.pressure

3. Systemd-oomd vs. Kernel OOM Killer

In many modern distributions (like Ubuntu 22.04+ and Fedora), the traditional kernel OOM killer has been augmented—or preempted—by systemd-oomd.

systemd-oomd operates in userspace and proactively monitors PSI. Instead of waiting for the system to completely run out of memory (memory.max), systemd-oomd will ruthlessly kill an entire cgroup if its memory pressure (memory.pressure > some or full threshold) remains too high for too long.

Diagnosing Which OOM Killer Acted

If your container dies, first check dmesg or the kernel journal:

journalctl -k | grep -i oom

If the kernel OOM killer acted, you will see a massive stack trace detailing oom_memcg (the cgroup that ran out of memory) and the specific process killed.

If the kernel log is empty, check systemd-oomd:

journalctl -u systemd-oomd

You might see log entries like:

systemd-oomd[641]: Killed /system.slice/docker-123abc...scope due to memory pressure for /system.slice/docker.service being 65.4% > 50.0% for > 20s.

This confirms that systemd-oomd killed the container proactively due to PSI stalls, not because it hit a hard memory limit.

4. Root Cause Analysis: Checking memory.events

When diagnosing a container memory issue, the most valuable file is memory.events within the container’s cgroup directory.

cat ${CGROUP_PATH}/memory.events

Output:

low 0
high 142
max 0
oom 0
oom_kill 0
oom_group_fault 0

How to interpret this:

• If high is incrementing rapidly but oom is 0, your container is constantly hitting its soft limit, getting throttled, and suffering terrible performance latency (High PSI), but surviving.
• If oom_kill is greater than 0, the container breached memory.max and a process was terminated by the kernel.

5. Mitigation Strategies & Best Practices

A. Tune memory.high Gracefully

Do not just set a hard memory limit in Docker (-m 2g). This sets memory.max to 2GB but leaves memory.high unset. Consequently, the container runs perfectly until it hits 2GB, at which point it violently crashes.

Instead, configure a memory.high limit slightly below your memory.max to allow the kernel to gently throttle the container and give the application time to garbage collect.

In docker run:

docker run --memory="2g" --memory-reservation="1.5g" ...

(Note: Docker translates --memory-reservation to memory.low, but you can use cgroup-parent or systemd drop-ins to explicitly set memory.high.)

B. Use memory.oom.group to Prevent Zombie Containers

By default, the OOM killer terminates the single process consuming the most memory. In a multi-process container (e.g., PHP-FPM and Nginx), it might kill one worker but leave the container running in a broken state.

Enable cgroup-wide OOM kills so the entire container stops and allows Docker’s restart policies to cleanly reboot it:

echo 1 > ${CGROUP_PATH}/memory.oom.group

C. Bypass Hardware Limits with Bare-Metal Servers

If you are constantly tweaking memory.high and battling systemd-oomd due to high PSI, you are simply masking a fundamental resource shortage. Heavy relational databases (MySQL/PostgreSQL), massive ELK stacks, and enterprise Java applications running in Docker are notoriously hostile to shared kernel environments and virtualized memory overhead.

When workloads mature beyond the capabilities of shared VMs or standard VPS hosting, the ultimate solution is escaping virtualization overhead entirely. Migrating high-IOPS, memory-intensive Docker Swarm or Kubernetes clusters to bare-metal Dedicated Servers eliminates hypervisor memory translation latency and guarantees 100% hardware exclusivity. For enterprises requiring regional edge performance and strict data sovereignty in South Asia, deploying on top-tier Dedicated Servers in Pakistan ensures that massive in-memory databases operate with zero PSI stalls, sub-millisecond TTFB, and absolute stability.

D. Adjust systemd-oomd Configuration

If systemd-oomd is too aggressive, you can increase the pressure threshold or duration before it acts. Edit /etc/systemd/oomd.conf:

[OOM]
DefaultMemoryPressureLimit=80%
DefaultMemoryPressureDurationSec=30s

Apply changes:

systemctl restart systemd-oomd

Conclusion

Troubleshooting OOM events in a cgroup v2 environment requires looking beyond just the hard memory limit. By understanding the relationship between memory.high throttling, Pressure Stall Information (PSI), and user-space daemons like systemd-oomd, you can accurately diagnose whether your containers are starving for resources, suffering from inefficient memory allocation, or proactively being killed by the OS.