Previous Post Recap

In the previous post, I confirmed that splitting the Debezium embedded engine’s callback into a submit thread (TRD1) and an order-preserving drain thread (TRD2) lets you process in parallel with no performance loss (98-101%) while still preserving order. But that was a pure benchmark - just a single id pulled out, queued, and checked for order.

This post set out to build two things.

1. Can you swap MySQL for Postgres without changing a single line of our code? Just as swapping a JDBC driver leaves the application logic untouched, I implemented Debezium so it’s only responsible for “where to capture from” - everything after that (transforming, sending) is application code we write however we need.
2. Does “parallel processing while preserving order,” confirmed by the benchmark, still hold when the workers are running a real transform and sending to a real Kafka, not just an id? Last time I measured order with a single id; this time I reproduced that same conclusion with a small application that transforms real columns and sends to a real Kafka.

The Full Data Flow

Multi-source only branches at the Debezium connector stage. Everything after that (TRD1 - worker pool - TRD2 - KafkaSink) is handled by exactly the same code regardless of source, and it all lands in the same topic.

CDC and Debezium

When you need to sync data in real time between heterogeneous systems (RDBMS, search engines, caches, other services’ databases, etc.), periodically polling the source database has limits in both scalability and latency. CDC (Change Data Capture) is the standard way to solve this — reading a database’s change log in real time and turning those changes into events delivered to wherever they’re needed. Debezium is one of the most widely used open-source tools for implementing CDC, and it’s a common core component in data-sync pipelines across heterogeneous systems.