In enterprise PostgreSQL deployments, high-volume e-commerce transaction engines, and multi-tenant SaaS backends running on Linux and cPanel/WHM environments, the Write-Ahead Log (WAL) subsystem is the fundamental guarantor of ACID compliance, point-in-time recovery (PITR), and streaming replication.[1] Before any data modification is permanently committed to table data files (base/), PostgreSQL writes the transaction sequence to sequential 16MB log files within the pg_wal/ directory (historically pg_xlog/ in versions prior to PostgreSQL 10).[1]
Under optimal production parameters, the PostgreSQL checkpointing background process regularly recycles or removes obsolete WAL segments once their transactions have been flushed to disk and archived.[1][2] However, when an upstream failure occurs—such as an abandoned standby server holding an active replication slot, a failing archive_command shell script, or insufficient disk headroom—PostgreSQL is strictly forbidden by design from deleting older WAL segments.[1][3]
Within hours, thousands of uncollected 16MB WAL files saturate the storage volume to 100% capacity.[1] The database server abruptly crashes with a fatal kernel-level panic:[1]
2026-11-12 10:14:22.812 UTC [28491] PANIC: could not write to file "pg_wal/xlogtemp.28491": No space left on device
2026-11-12 10:14:23.014 UTC [28489] LOG: checkpointer process (PID 28491) was terminated by signal 6: Aborted
2026-11-12 10:14:23.015 UTC [28489] LOG: terminating any other active server processes
2026-11-12 10:14:23.019 UTC [28489] LOG: all server processes were terminated; reinitializing
2026-11-12 10:14:23.102 UTC [28540] FATAL: the database system is in recovery mode
2026-11-12 10:14:23.150 UTC [28540] PANIC: could not locate a valid checkpoint recordWhen confronted with a crashed server and a 100% full partition, inexperienced administrators frequently make a catastrophic error: manually executing rm -rf pg_wal/*.[1][2] Deleting uncheckpointed WAL files permanently destroys the database cluster’s internal transaction state, forcing an unrecoverable restoration from cold backups.[1][2]
This diagnostic runbook provides systems administrators, site reliability engineers (SREs), and database engineers with a safe, proven procedure to triage disk exhaustion, create emergency storage headroom, identify culprit replication slots or archive failures, and recover PostgreSQL services without data loss.[1][4]
Architectural Foundations: What Prevents WAL Segment Recycling?
PostgreSQL dynamically manages WAL files between min_wal_size and max_wal_size.[1] In a standalone server without archiving, PostgreSQL renames old segments to future sequence numbers rather than deleting them, minimizing filesystem allocation overhead.[1]
However, three architectural mechanisms override checkpoint recycling, forcing PostgreSQL to retain every generated WAL file indefinitely:[1][3]
+--------------------------------------------------------------------------+
| PostgreSQL Write-Ahead Logging Retention Subsystem |
+--------------------------------------------------------------------------+
| | |
v v v
+-----------------------+ +--------------------+ +--------------+
| Physical / Logical | | Continuous Archive | | High-Volume |
| Replication Slots | | Command Failures | | WAL Burst |
| (restart_lsn hold) | | (Non-zero exit) | | (Under-sized)|
+-----------------------+ +--------------------+ +--------------+
| | |
v v v
Standby replica goes down; S3 / NFS / Backup script max_wal_size
primary preserves all WALs to fails; PostgreSQL refuses exhausted before
allow replica to reconnect to delete unarchived files checkpoint completes1. Inactive or Lagging Replication Slots
Physical and logical replication slots ensure that a standby replica never misses changes.[1][3] A slot maintains a pointer (restart_lsn) representing the earliest WAL record required by the standby.[1] If a standby instance is decommissioned, powered down, or encounters network isolation without explicitly dropping its slot, the primary cluster retains every single WAL segment generated since the disconnect.[1][3] In PostgreSQL configurations where max_slot_wal_keep_size is left at its default value (-1), the primary will fill 100% of the disk and crash rather than abandon the replica.[1]
2. Failing Continuous Archiving (archive_command)
When continuous archiving (archive_mode = on) is enabled, PostgreSQL invokes archive_command (e.g. executing pgbackrest, wal-g, or an rsync script) for each completed 16MB segment.[1] If the remote backup destination fills up, network authentication expires, or the command returns an exit code other than 0, PostgreSQL considers the file unarchived and retries indefinitely.[1] Unarchived segments cannot be purged, accumulating until disk space is fully exhausted.[1]
3. Unconstrained wal_keep_size
The wal_keep_size setting specifies the minimum volume of WAL files kept in pg_wal to allow streaming replicas to catch up without slots.[1] Setting this parameter too high on a shared storage partition can starve the base data directory of writable blocks.[1]
Failure Triage Matrix
| Failure Symptom | Underlying Root Cause | Verification Command | Permanent Solution |
|---|---|---|---|
PANIC: could not write to file ... No space left |
pg_wal partition reached 100% disk usage.[1][5] |
df -h /var/lib/postgresql/ |
Create emergency space, inspect pg_controldata, start database, force checkpoint.[1][4][6] |
Thousands of .ready files in pg_wal/archive_status |
archive_command failing repeatedly; non-zero exit code.[1] |
SELECT * FROM pg_stat_archiver; |
Fix backup target permissions or temporarily set archive_command = '/bin/true'.[1] |
| Database running out of disk while traffic is moderate | Abandoned replication slot holding restart_lsn.[1][3] |
SELECT * FROM pg_replication_slots; |
Drop dead slot via pg_drop_replication_slot(); configure max_slot_wal_keep_size.[1][3] |
Crash loop on startup: FATAL: database in recovery |
Server cannot write recovery checkpoint to full disk.[1] | systemctl status postgresql |
Reclaim emergency blocks using tune2fs -m 0 or delete crash dumps.[1][6] |
Emergency Recovery Runbook: Resolving Disk Exhaustion Safely
When PostgreSQL has crashed with No space left on device, follow this exact sequence to restore operations without destroying data integrity.[1][4]
Phase 1: Create Emergency Disk Headroom (DO NOT touch pg_wal)
PostgreSQL cannot perform crash recovery or start up without writing a small amount of metadata.[1] You must free up disk space elsewhere on the partition first:[5][6]
1. Reclaim Linux Ext4 Reserved Blocks
By default, Linux Ext4 filesystems reserve 5% of storage blocks for the root user.[6] You can instantly release this reserved space to allow the postgres user to write temporary recovery blocks:[6]
# Identify the full filesystem device
df -h /var/lib/postgresql
# Reduce reserved blocks from 5% to 0% (provides immediate gigabytes of headroom)
tune2fs -m 0 /dev/mapper/ubuntu--vg-ubuntu--lv2. Remove Rotated System Logs and Core Dumps
Check for historical compressed log files or crash dumps on the partition:[5]
# Remove rotated compressed logs older than 7 days
find /var/log/postgresql/ -name "*.gz" -delete
# Remove core dumps if present
rm -f /var/lib/postgresql/core.*3. Expand the LVM Volume (If Available)
If the server utilizes Logical Volume Manager (LVM) with unallocated physical extent storage, expand the volume and resize the filesystem online:[5]
lvextend -L +10G /dev/mapper/ubuntu--vg-ubuntu--lv -rPhase 2: Inspect Cluster Integrity with pg_controldata
Before attempting to start the database service, inspect the control file to verify the database state and locate the latest checkpoint location:[4]
su - postgres
/usr/lib/postgresql/16/bin/pg_controldata /var/lib/postgresql/16/mainExamine the output for two critical indicators:[4]
- Database cluster state: Should report
in crash recoveryorshut down in recovery.[4] - Latest checkpoint’s REDO WAL file: e.g.,
00000001000004120000008A. This confirms which WAL segment is currently required for crash recovery.[1][4]
Phase 3: Start PostgreSQL with Protective Overrides
To prevent PostgreSQL from crashing again immediately upon boot due to a looping archive_command, start the instance with archiving temporarily disabled or bypassed:[1]
- Edit
/etc/postgresql/16/main/postgresql.conf(or/var/lib/postgresql/16/main/postgresql.auto.conf):[1]# Temporarily force archive_command to succeed to stop file accumulation archive_command = '/bin/true' - Start the PostgreSQL system service:[1]
systemctl start postgresql - Verify that PostgreSQL has completed crash recovery and is accepting queries:[1]
If it returnssu - postgres -c "psql -c 'SELECT pg_is_in_recovery();'"f(false), crash recovery succeeded, and the cluster is now operating as the primary read/write engine.[1]
Phase 4: Identify and Remove Culprit Replication Slots
Log into the psql console as superuser to audit active replication slots and identify who is holding back WAL truncation:[1][3]
SELECT
slot_name,
slot_type,
active,
wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal_bytes
FROM pg_replication_slots; slot_name | slot_type | active | wal_status | retained_wal_bytes
---------------+-----------+--------+------------+--------------------
standby_node2 | physical | f | unreserved | 48 GB
reporting_bi | logical | t | normal | 32 MB
(2 rows)In the output above, slot standby_node2 is inactive (active: f) and is holding back 48 GB of WAL segments.[1][3] Drop the abandoned slot immediately:[3]
SELECT pg_drop_replication_slot('standby_node2');Phase 5: Force Checkpoint and Reclaim Disk Space
Once the offending replication slot has been dropped and archive_command is bypassed or operational, trigger an immediate checkpoint to force PostgreSQL to prune the retained WAL directory:[1]
CHECKPOINT;Monitor the filesystem in another terminal console:[5]
watch -n 1 'df -h /var/lib/postgresql && ls -1 /var/lib/postgresql/16/main/pg_wal | wc -l'PostgreSQL will immediately remove or recycle the excess segments, restoring available storage headroom.[1]
Phase 6: Safe WAL Pruning via pg_archivecleanup (When Offline)
If PostgreSQL cannot boot because the partition is 100% full and no temporary space can be freed via OS tools, you must NEVER run arbitrary rm commands.[1][2] Instead, use the official PostgreSQL utility pg_archivecleanup.[2]
- Extract the latest checkpoint REDO file using
pg_controldata:su - postgres /usr/lib/postgresql/16/bin/pg_controldata /var/lib/postgresql/16/main | grep "REDO WAL file" # Example output: Latest checkpoint's REDO WAL file: 00000001000004120000008A - Run
pg_archivecleanupin dry-run mode (-d) to preview what can be safely purged:[2]/usr/lib/postgresql/16/bin/pg_archivecleanup -d /var/lib/postgresql/16/main/pg_wal 00000001000004120000008A - Execute the cleanup. This deletes only WAL files older than the REDO point, preserving cluster consistency:[2]
/usr/lib/postgresql/16/bin/pg_archivecleanup /var/lib/postgresql/16/main/pg_wal 00000001000004120000008A
Hardening postgresql.conf Against Future Disk Exhaustion
To prevent replication disconnects or archive failures from crashing production clusters, configure these critical guardrail directives in postgresql.conf:
# ----------------------------------------------------------------------
# 1. Protect Cluster from Inactive Standby Disconnects
# ----------------------------------------------------------------------
# In PostgreSQL 13+, limits how much WAL an inactive replication slot can retain.
# If a standby drops and retains more than 32GB, the primary invalidates the slot
# instead of allowing the disk to fill and crashing the cluster.
max_slot_wal_keep_size = 32GB
# ----------------------------------------------------------------------
# 2. Bound WAL Retention on Disk
# ----------------------------------------------------------------------
min_wal_size = 2GB
max_wal_size = 16GB
wal_keep_size = 4GB
# ----------------------------------------------------------------------
# 3. Smooth Checkpoint I/O
# ----------------------------------------------------------------------
# Spreads checkpoint writes over 90% of the checkpoint interval to prevent I/O spikes
checkpoint_completion_target = 0.9
checkpoint_timeout = 15min
# ----------------------------------------------------------------------
# 4. Archiving Safety Timeout
# ----------------------------------------------------------------------
# Forces a WAL file switch if no transactions have completed for 5 minutes,
# ensuring predictable archive shipping
archive_timeout = 300Apply the changes and reload PostgreSQL configuration without service disruption:[1]
SELECT pg_reload_conf();Storage Architecture: Separate WAL from Base Data
In enterprise database engineering, colocating pg_wal and base/ on the same storage partition introduces significant operational risk.[1] A single large table vacuum, temp file generation from unindexed sorting queries, or runaway WAL burst will compromise both transaction logging and table storage simultaneously.[1]
Systems architects operating mission-critical databases eliminate this single point of failure by deploying high-performance dedicated servers and localized dedicated servers in Pakistan. Enterprise bare-metal infrastructure enables the configuration of independent PCIe Gen4/Gen5 NVMe storage arrays—isolating pg_wal on a dedicated, ultra-low-latency array with dedicated write channels, ensuring high-throughput write-ahead logging completely insulated from base data table I/O.[1]
Sources & Official References
- PostgreSQL Global Development Group. (2026). PostgreSQL 18 Documentation: Reliability and the Write-Ahead Log (WAL Configuration and Retention). Retrieved from https://www.postgresql.org/docs/current/wal-configuration.html
- PostgreSQL Global Development Group. (2026). PostgreSQL 18 Documentation: pg_archivecleanup — Clean Up PostgreSQL WAL Archive Files. Retrieved from https://www.postgresql.org/docs/current/pgarchivecleanup.html
- PostgreSQL Global Development Group. (2026). PostgreSQL 18 Documentation: Streaming Replication Slots and max_slot_wal_keep_size Guardrails. Retrieved from https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS
- PostgreSQL Global Development Group. (2026). PostgreSQL 18 Documentation: pg_controldata — Display Control Information of a PostgreSQL Database Cluster. Retrieved from https://www.postgresql.org/docs/current/app-pgcontroldata.html
- Linux Programmer's Manual. (2026). df(1) - Report File System Disk Space Usage and Available Storage Inodes. Retrieved from https://man7.org/linux/man-pages/man1/df.1.html
- Linux Programmer's Manual. (2026). tune2fs(8) - Adjust Tunable Filesystem Parameters on Ext2/Ext3/Ext4 Filesystems (Reserved Blocks Management). Retrieved from https://man7.org/linux/man-pages/man8/tune2fs.8.html
Eliminate PostgreSQL I/O Bottlenecks with Dedicated Bare-Metal NVMe
Protect your transactional databases from disk saturation, WAL bottlenecks, and replication stalls. Deploy Nextgen's high-performance enterprise bare-metal dedicated servers with dedicated PCIe Gen4 NVMe arrays engineered specifically for zero-latency database performance.
Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.
