MariaDB Change Data Capture (CDC) with Debezium and Apache Kafka: Real-Time Event-Driven Replication for Enterprise Workloads in Pakistan

Implement high-throughput Change Data Capture (CDC) from MariaDB to Apache Kafka using Debezium, binary log row-based replication, and distributed streaming in Pakistan.

MariaDB Change Data Capture (CDC) with Debezium and Apache Kafka: Real-Time Event-Driven Replication for Enterprise Workloads in Pakistan

Modern enterprise applications across Pakistan—spanning real-time payment processing gateways, fintech ledgers, large-scale supply chain logistics, and high-frequency e-commerce portals—require immediate visibility into database changes. Relying on polling-based architectures (such as querying updated_at > NOW() - INTERVAL 1 MINUTE) introduces devastating database lock contention, wasted I/O bandwidth, and intolerable data propagation delays.

Change Data Capture (CDC) solves this architectural challenge by non-intrusively tailing the database’s native binary commit log (binlog) at line rate. By integrating MariaDB with Debezium and Apache Kafka, engineering teams can capture row-level INSERT, UPDATE, and DELETE mutations with sub-10ms delivery latency, streaming events directly into search indexes (Elasticsearch/OpenSearch), analytical caches (Redis), and data warehouses without impacting transaction throughput.


The Architecture of Binlog-Based CDC

Traditional MariaDB replication streams the binary log to replica servers. Debezium operates as a specialized replication client that registers itself as a pseudo-replica with its own unique server-id.

MariaDB Change Data Capture (CDC) with Debezium and Apache Kafka: Real-Time Event-Driven Replication for Enterprise Workloads in Pakistan Architecture & Workflow

When operating on mission-critical Dedicated Servers in Pakistan, hosting MariaDB and the Kafka broker on bare-metal hardware with direct-attached NVMe arrays ensures that high-volume CDC event ingestion does not cause binary log serialization stalls or disk queue exhaustion.


Step 1: Configuring MariaDB for Row-Based Replication (RBR)

For Debezium to reconstruct the exact “before” and “after” state of mutated rows, MariaDB must be configured with Row-Based Replication (binlog_format=ROW) and full row imaging.

Edit /etc/my.cnf.d/server.cnf under the [mariadb] section:

[mariadb]
# Enable binary logging with unique server ID
server-id = 10101
log_bin = /var/lib/mysql/mariadb-bin
log_bin_index = /var/lib/mysql/mariadb-bin.index
expire_logs_days = 7
max_binlog_size = 1G

# Critical CDC requirements
binlog_format = ROW
binlog_row_image = FULL

# Enable GTID for deterministic failover and offset tracking
gtid_strict_mode = ON

# Ensure transactional integrity
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1

Restart the MariaDB service and verify the settings:

systemctl restart mariadb
mysql -u root -p -e "SHOW VARIABLES LIKE 'binlog_format';"

Step 2: Provisioning Dedicated CDC Service Credentials

Create an isolated database user with strict permissions required to read the binlog stream without granting broad administrative privileges:

-- Create Debezium CDC user
CREATE USER 'debezium_cdc'@'10.0.0.%' IDENTIFIED BY 'SuperSecureCDCPassword2026!';

-- Grant required replication and metadata privileges
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT 
ON *.* TO 'debezium_cdc'@'10.0.0.%';

-- Grant lock tables permission for initial schema snapshots
GRANT LOCK TABLES ON *.* TO 'debezium_cdc'@'10.0.0.%';

FLUSH PRIVILEGES;

Step 3: Deploying the Debezium Connector on Kafka Connect

Deploy the Debezium connector by submitting a JSON specification to the Kafka Connect REST API (running on port 8083):

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" \
  http://kafka-connect.internal.pk:8083/connectors/ \
  -d '{
    "name": "mariadb-cdc-inventory-connector",
    "config": {
      "connector.class": "io.debezium.connector.mysql.MySqlConnector",
      "tasks.max": "1",
      "database.hostname": "10.0.0.50",
      "database.port": "3306",
      "database.user": "debezium_cdc",
      "database.password": "SuperSecureCDCPassword2026!",
      "database.server.id": "99901",
      "topic.prefix": "pakistan_fintech",
      "database.include.list": "fintech_db",
      "table.include.list": "fintech_db.transactions,fintech_db.settlements",
      "schema.history.internal.kafka.bootstrap.servers": "kafka1.internal.pk:9092,kafka2.internal.pk:9092",
      "schema.history.internal.kafka.topic": "schema-changes.fintech_db",
      "database.ssl.mode": "required",
      "snapshot.mode": "initial",
      "tombstones.on.delete": "true"
    }
  }'

Step 4: Consuming Real-Time CDC Mutation Events

Once the connector transitions to RUNNING, Debezium publishes events to Kafka topics following the convention: <topic.prefix>.<database>.<table>. Each message payload contains the comprehensive transaction metadata:

{
  "before": {
    "transaction_id": 482910,
    "user_id": 1402,
    "amount": 2500.00,
    "status": "PENDING"
  },
  "after": {
    "transaction_id": 482910,
    "user_id": 1402,
    "amount": 2500.00,
    "status": "SETTLED"
  },
  "source": {
    "version": "2.7.0.Final",
    "connector": "mysql",
    "name": "pakistan_fintech",
    "ts_ms": 1790931200150,
    "db": "fintech_db",
    "table": "transactions",
    "server_id": 10101,
    "gtid": "0-10101-948210"
  },
  "op": "u",
  "ts_ms": 1790931200165
}

The op field indicates the operation (c for create, u for update, d for delete), giving downstream microservices deterministic, change-aware event triggers.


Diagnosing Binlog Lag and Connector Health

To verify connector throughput and ensure zero binlog replication lag:

# Check Kafka Connect task status
curl -s http://kafka-connect.internal.pk:8083/connectors/mariadb-cdc-inventory-connector/status | jq .

# Monitor MariaDB binlog disk consumption
mysql -u root -p -e "SHOW BINARY LOGS;"

Deploying real-time event streaming architectures on bare-metal Dedicated Servers provides dedicated NVMe write throughput, multi-gigabit private interconnects, and unmetered network bandwidth to prevent I/O serialization bottlenecks across enterprise MariaDB clusters.

Need Enterprise Dedicated Infrastructure in Pakistan?

Deploy mission-critical, bare-metal infrastructure optimized for low-latency throughput, hardware RAID/NVMe resilience, and 24/7 proactive management.

Disclaimer: This post is for informational and educational purposes only, accurate to the best of our knowledge at publication. It does not constitute advice. We do not guarantee accuracy or completeness and accept no liability for any decisions or losses resulting from reliance on this content.