Diagnosing OOM Kills in MySQL 8.0: Performance Schema and InnoDB Buffer Pool Sizing

Database administrators running MySQL 8.0 on high-throughput Linux environments often encounter sudden, unexplained database restarts. A quick inspection of /var/log/syslog or dmesg frequently reveals a familiar culprit: the Linux Out-Of-Memory (OOM) killer has terminated the mysqld process.

While the immediate reaction might be to increase server RAM, true resolution requires diagnosing why MySQL 8.0 is consuming more memory than allocated. This article walks through advanced diagnostic techniques using the Performance Schema, kernel log analysis, and systemd overrides to stabilize your database environment.

The Symptoms: Catching the OOM Killer in Action

When a database becomes unreachable and restarts, the first step is verifying if the Linux kernel’s OOM killer intervened. Use dmesg or journalctl to search for OOM events:

dmesg -T | egrep -i 'killed process'

Alternatively, search the kernel logs:

grep -i "Out of memory" /var/log/messages
# or on Debian/Ubuntu systems:
grep -i "Out of memory" /var/log/syslog

You might see an output similar to this:

[Tue Sep 15 14:32:12 2026] mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Tue Sep 15 14:32:12 2026] CPU: 4 PID: 18324 Comm: mysqld Tainted: G    B   W         5.15.0-101-generic #111-Ubuntu
[Tue Sep 15 14:32:12 2026] Call Trace:
[Tue Sep 15 14:32:12 2026]  <TASK>
...
[Tue Sep 15 14:32:12 2026] Out of memory: Killed process 18324 (mysqld) total-vm:18943212kB, anon-rss:15821004kB, file-rss:0kB, shmem-rss:0kB, UID:114 pgtables:33812kB oom_score_adj:0

The key metric here is anon-rss:15821004kB, which indicates mysqld was holding nearly 15.8 GB of anonymous memory resident in RAM when it was terminated. If your system only has 16GB of RAM, the OOM killer’s action was unavoidable.

Phase 1: Reviewing Basic Memory Allocation (The Buffer Pool)

In MySQL, the largest single memory consumer is typically the innodb_buffer_pool_size. A common rule of thumb is setting this to 60-80% of total available RAM on a dedicated database server. However, MySQL’s total memory footprint is the buffer pool plus various global buffers and per-connection buffers.

If you have a 16GB server and set the buffer pool to 12GB (75%), but also have max_connections = 1000 with large per-thread buffers (e.g., sort_buffer_size, join_buffer_size), simultaneous complex queries can easily push total consumption past 16GB.

Check your current settings:

SELECT @@innodb_buffer_pool_size / 1024 / 1024 / 1024 AS 'Buffer_Pool_GB',
       @@max_connections AS 'Max_Connections';

If your per-thread buffers are excessively large, throttle them down in your my.cnf:

[mysqld]
# Ensure buffer pool leaves room for the OS and other processes
innodb_buffer_pool_size = 8G
max_connections = 300
sort_buffer_size = 2M
join_buffer_size = 2M

Phase 2: MySQL 8.0 Memory Leaks and Performance Schema

If the basic math checks out but mysqld memory continuously climbs until OOM, you may be facing a memory leak or unexpected allocation. MySQL 8.0 introduced sophisticated memory tracking via the Performance Schema.

Enabling Memory Tracking

Ensure Performance Schema memory instrumentation is enabled. In your my.cnf:

[mysqld]
performance_schema = ON
performance_schema_instrument = 'memory/%=COUNTED'

Restart MySQL (if you changed the config), and then run this query to aggregate memory usage by event name:

SELECT event_name, 
       current_alloc, 
       high_alloc 
FROM sys.memory_global_by_current_bytes 
WHERE current_alloc > '10 MiB' 
LIMIT 10;

Analyzing the Output

A healthy system will show the InnoDB buffer pool at the top:

+-----------------------------------------------------------------------------+---------------+-------------+
| event_name                                                                  | current_alloc | high_alloc  |
+-----------------------------------------------------------------------------+---------------+-------------+
| memory/innodb/buf_buf_pool                                                  | 8.00 GiB      | 8.00 GiB    |
| memory/innodb/os0file                                                       | 152.12 MiB    | 180.55 MiB  |
| memory/performance_schema/events_statements_history_long                    | 76.80 MiB     | 76.80 MiB   |
| memory/performance_schema/events_statements_history_long.sql_text           | 48.00 MiB     | 48.00 MiB   |
+-----------------------------------------------------------------------------+---------------+-------------+

However, if you see excessive memory held by dictionary caches (memory/innodb/dict_stats_bg_recalc_pool_t or memory/sql/Table_cache::m_table_cache), you might be dealing with an excessive number of tables or a known bug in table definition cache handling.

Similarly, high allocations in memory/temptable/physical_ram indicate that temporary tables (created for complex GROUP BY or ORDER BY operations) are consuming vast amounts of RAM. In MySQL 8.0, the TempTable engine replaced MEMORY for internal temporary tables. You can control its memory limit:

# Limit TempTable memory before it spills to disk (default is 1G)
temptable_max_ram = 512M

Phase 3: Systemd and OOMScoreAdjust

Sometimes, during backups or intensive batch jobs, other system processes (like gzip or rsync) might spike memory usage, causing the OOM killer to indiscriminately target the largest process—which is always mysqld.

To protect MySQL, you can adjust its OOM score via systemd.

1. Open the systemd override file for MySQL:

   systemctl edit mysql

2. Add the following lines to make MySQL less favorable to the OOM killer:

   [Service]
   OOMScoreAdjust=-800

   (Note: Valid values range from -1000 to 1000. -1000 completely disables OOM killing for the process, which is generally not recommended as it can cause kernel panics. -800 provides strong protection).

3. Reload systemd and restart MySQL:

   systemctl daemon-reload
   systemctl restart mysql

Scaling Up: When Tuning Isn’t Enough

Sometimes, aggressive tuning and memory profiling confirm that your database’s active dataset and concurrency simply require more physical memory than your current environment can provide. Virtual instances often share memory resources or impose tight cgroup constraints that make resource isolation difficult under heavy load.

To bypass shared memory limits, avoid noisy-neighbor virtualization penalties, and prevent MySQL OOM kills permanently, the most reliable solution is migrating heavy database nodes to bare-metal hardware. Upgrading to robust Dedicated Servers provides unshared RAM, dedicated CPU cache, and superior disk I/O—crucial for large InnoDB buffer pools. If your primary user base is in South Asia, deploying your database tier on low-latency Dedicated Servers in Pakistan can dramatically improve query response times while ensuring your mysqld process has the guaranteed, exclusive hardware resources it needs to remain stable under massive concurrent load.

Conclusion

OOM kills in MySQL 8.0 are rarely a mystery if you know where to look. By checking kernel logs to confirm the OOM event, auditing global and per-thread buffers, utilizing the Performance Schema to track memory allocations down to the subsystem level, and adjusting systemd protections, you can stabilize your database. And when software tuning reaches its limits, scaling out to dedicated hardware remains the definitive cure.