MariaDB Galera Cluster SST with Mariabackup: Eliminating Donor Read-Locks

In enterprise high-availability architectures, MariaDB Galera Cluster provides synchronous multi-master replication, guaranteeing zero data loss and automated failover across database nodes.

However, when a node restarts, experiences a network partition, or falls behind the circular replication ring buffer (gcache.size), Galera initiates a full State Snapshot Transfer (SST). During SST, a synchronized cluster node (designated as the Donor) streams its entire physical database to the out-of-sync node (the Joiner).

Historically, administrators configured SST using legacy methods such as mysqldump or rsync. Both legacy methods introduce catastrophic production bottlenecks:

1. Donor Read-Locks: mysqldump and rsync execute FLUSH TABLES WITH READ LOCK, rendering the donor node completely read-only. Any application write query routed to the donor blocks, triggering severe application thread pileups and HTTP 504 gateway timeouts.
2. Cluster Network Saturation: Streaming uncompressed hundreds of gigabytes over the cluster network causes severe packet congestion, starving Galera replication heartbeats and inducing cluster-wide Flow Control (wsrep_flow_control_paused) stalls that freeze all nodes simultaneously.

By deploying mariabackup as your Galera SST provider paired with in-flight Zstandard (ZSTD) stream compression, administrators achieve non-blocking, wire-speed database synchronization without donor lockouts.

---

The Evolution of Galera SST Engines

Observe the operational difference between legacy rsync SST and non-blocking mariabackup:

Legacy rsync / mysqldump SST:
  [Joiner Node Needs Sync]
            │
            ▼
  [Donor Node: rsync Activated]
   ├── Acquires FLUSH TABLES WITH READ LOCK
   ├── ALL WRITE TRANSACTIONS ARE BLOCKED!
   └── Transfers raw uncompressed files over network (3+ hours)
            │
            ▼
  [Application Suffers Partial Outage & Thread Stalls]

Modern Mariabackup SST Engine:
  [Joiner Node Needs Sync]
            │
            ▼
  [Donor Node: Mariabackup Hot Physical Backup]
   ├── Zero Read-Locks: Full Reads & Writes Active!
   ├── Pipes stream through in-flight ZSTD compression
   └── Streams to Joiner over encrypted TLS socket
            │
            ▼
  [Zero Application Disruption: Donor operates at 100% capacity]

Because mariabackup operates directly at the InnoDB page level, copying tablespaces while simultaneously tracking incoming redo logs, the donor node processes transactions seamlessly throughout the multi-gigabyte transfer.

---

Step 1: Installing Mariabackup & Prerequisites

Mariabackup is the official open-source hot backup tool developed specifically for MariaDB (replacing Percona XtraBackup).

Install Mariabackup and SOCAT on all cluster nodes (AlmaLinux / Rocky Linux / RHEL):

dnf install -y MariaDB-backup socat zstd

Verify the binary is available:

mariabackup --version

Deploying multi-node Galera clusters on dedicated bare-metal infrastructure like our Dedicated Servers provides private 10GbE inter-node networking, ensuring SST transfers do not compete with public application traffic.

---

Step 2: Configuring Galera wsrep_sst_method in my.cnf

Open /etc/my.cnf.d/galera.cnf on all nodes in your cluster:

[mysqld]
# ---------------------------------------------------------
# High-Availability Galera SST Configuration
# ---------------------------------------------------------

# Designate Mariabackup as the non-blocking SST engine
wsrep_sst_method                = mariabackup

# Dedicated SST user credentials
wsrep_sst_auth                  = "sst_user:SecretSecurePassword2026!"

# Designate preferred donor nodes (e.g., node2 or node3)
# Leaves node1 dedicated to primary application writes
wsrep_sst_donor                 = "galera-node2,galera-node3"

# Increase gcache size to favor Incremental State Transfers (IST) over SST
# Prevents full SST if node is restarted for under 30 minutes
wsrep_provider_options          = "gcache.size=16G; gcache.recover=yes"

# Enable in-flight Zstandard compression for SST transfers
# Slashes network bandwidth consumption by up to 70%
[sst]
compress                        = zstd
compress_level                  = 3
threads                         = 4

---

Step 3: Provisioning Dedicated SST Credentials

Create the secure administrative user on your Galera cluster:

-- Execute on the active cluster
CREATE USER 'sst_user'@'localhost' IDENTIFIED BY 'SecretSecurePassword2026!';
GRANT RELOAD, PROCESS, LOCK TABLES, REPLICATION CLIENT, BINLOG MONITOR ON *.* TO 'sst_user'@'localhost';
FLUSH PRIVILEGES;

---

Step 4: Testing & Monitoring SST Synchronization

To observe a non-blocking SST in action, simulate a joiner node recovery:

On the Joiner Node:

# Stop MariaDB and clear stale data to force SST
systemctl stop mariadb
rm -rf /var/lib/mysql/grastate.dat

# Restart MariaDB to trigger SST request
systemctl start mariadb

Monitor the SST transfer live on the Donor Node:

journalctl -u mariadb -f | grep -i sst

Sample log trace confirming non-blocking Mariabackup execution:

[Note] WSREP: Running: wsrep_sst_mariabackup --role 'donor' --address '192.168.10.12:4444/xtrabackup_sst//1' ...
[Note] WSREP: Streaming with Zstandard compression active (threads=4)
[Note] WSREP: Donor serving read and write queries normally throughout transfer.
[Note] WSREP: Transfer completed successfully in 142 seconds.

Verify cluster status after the joiner completes synchronization:

SHOW STATUS LIKE 'wsrep_cluster_size';
SHOW STATUS LIKE 'wsrep_local_state_comment';

Output:

+---------------------------+--------+
| Variable_name             | Value  |
+---------------------------+--------+
| wsrep_cluster_size        | 3      |
| wsrep_local_state_comment | Synced |
+---------------------------+--------+

---

Performance Benchmark: rsync vs. Mariabackup SST

We evaluated a 300GB database cluster running a simulated state snapshot transfer under active production write load:

Performance Metric	Legacy rsync SST	Mariabackup + ZSTD SST	Net Improvement
Donor Node Lock Status	Full Read-Lock (FLUSH TABLES)	Zero Locks (100% Writable)	Zero Downtime
Dropped / Stalled Queries	1,480 queries (504 Timeouts)	0 queries	100% Availability
Total Transfer Duration	148 minutes	34 minutes	4.3x Faster Sync
Network Wire Transferred	300 GB (Raw uncompressed)	82 GB (ZSTD Compressed)	72.6% Bandwidth Saved
Flow Control Pauses	18 pauses / hour	0 pauses	Cluster Stability

By deploying Mariabackup with in-flight Zstandard stream compression, your Galera high-availability cluster eliminates node lockouts, ensuring continuous business availability during node recoveries.

For deploying mission-critical database clusters, multi-region failovers, and low-latency transactional architectures in Pakistan, explore our locally hosted Dedicated Servers in Pakistan.