Change data capture (CDC)
Also known as: CDC, log-based change data capture
Change data capture (CDC) is a technique for detecting every change made to a database, such as new rows, updates, and deletions, and passing those changes to other systems almost immediately.
It's how a warehouse, search index, or cache stays in sync with an operational database without heavy nightly copies. While CDC sounds like a niche data engineering trick, it's become the backbone of real-time analytics and event-driven systems.
Change data capture (CDC), explained at three levels
At home
Think of a shared family calendar. Instead of re-copying the entire calendar every night, someone texts everyone the moment an event is added, moved, or cancelled. Everyone stays up to date and nobody re-reads the whole thing. Change data capture does this for databases: it detects each change as it happens and sends just that change to other systems.
At work
Change data capture, or CDC, detects every insert, update, and delete in a database and passes those changes to other systems within seconds. It's how a warehouse, search index, or cache stays in sync with an operational database without heavy nightly copies. Businesses care because reports reflect what's happening now, and the main database carries far less load than when whole tables are copied repeatedly.
At home
Databases keep a log of every change, written before the change is applied, so they can recover after a crash. PostgreSQL calls it the write-ahead log, MySQL the binary log. Log-based CDC reads that log as it grows and turns each entry into an event, such as 'order 1234 status changed from pending to shipped,' which other systems receive in the same order the changes happened.
At work
There are three main approaches. Log-based CDC reads the transaction log and is the most complete and efficient, capturing deletes with little load. Trigger-based CDC uses database triggers to copy changes into a separate table, adding write overhead. Query-based polling looks for rows with recent timestamps, which is simple but misses deletes. Tools such as Debezium, AWS Database Migration Service, and Google Datastream publish the resulting events to Kafka or cloud streams.
At home
CDC has to start somewhere. It first takes a snapshot, a full copy of existing data, then switches to streaming changes from the log, and the two must line up exactly so nothing is missed or counted twice. Debezium handles this by recording the log position when the snapshot begins. In my opinion, that handover is the hardest part to get right, and it's where homemade CDC scripts usually fail.
At work
Production CDC must handle schema changes in source tables, often with a schema registry; ordering, which is usually guaranteed only per key or partition; and duplicates, since most pipelines deliver at least once and need idempotent consumers. Logs must be retained long enough for connectors to catch up after outages, but PostgreSQL replication slots hold logs until they're consumed, so a stalled connector can fill the source's disk. CDC streams carry personal data, so filter and mask early.
Real-world examples
At home
Lena built a side project that showed live inventory for a friend's bike shop. Using Debezium on the shop's PostgreSQL database, the website updated within seconds whenever a bike was sold in the store.
At work
Marcus is a data engineer at a Canadian online pharmacy. Nightly copies meant dashboards were always a day behind. Switching to CDC let the operations team see order delays in near real time and reroute shipments the same afternoon.
How it compares
| feature | Change data capture (CDC) | Batch extracts |
|---|---|---|
| Freshness | Seconds | Hours to a day |
| Load on source | Low, reads logs | High during copies |
| Captures deletes | Yes, with log-based CDC | Often not |
| Complexity | Higher | Lower |
CDC trades simplicity for fresh data and lighter database load.
For analysts
Before paying for real-time data, ask who will act on it within minutes. I'd use CDC where faster data changes a decision, such as fraud, inventory, or logistics, and keep simpler batches elsewhere.
History & origin
Databases have long kept transaction logs for crash recovery and replication, and commercial replication tools such as Oracle GoldenGate (acquired by Oracle in 2009) used them to copy changes between systems. LinkedIn's work on Databus and later Apache Kafka in the early 2010s showed how changes could flow as streams to many consumers. Red Hat launched the open-source Debezium project around 2016, which read logs from MySQL, PostgreSQL, and others and published changes to Kafka. Cloud services, including AWS Database Migration Service and Google Datastream, made log-based CDC a managed feature. In my view, CDC finally killed the 'nightly batch' as the default, and good riddance.