CDC and Debezium

When you need to sync data in real time between heterogeneous systems (RDBMS, search engines, caches, other services’ databases, etc.), periodically polling the source database has limits in both scalability and latency. CDC (Change Data Capture) is the standard way to solve this — reading a database’s change log in real time and turning those changes into events delivered to wherever they’re needed. Debezium is one of the most widely used open-source tools for implementing CDC, and it’s a common core component in data-sync pipelines across heterogeneous systems.

There comes a point where a single MySQL instance just can’t keep up with the data anymore. Vitess is one of the approaches YouTube, Slack, and GitHub have used to solve this problem. I worked through what Vitess actually does, what components it’s made of, and how those components should actually be laid out, by building it myself in a local Docker environment. The setup shown here is posted as-is in a GitHub repo, and you can spin it up yourself with nothing but Docker.