Change data capture (CDC)
Also known as: CDC, log-based change data capture
Change data capture (CDC) is a technique for detecting every change made to a database, such as new rows, updates, and deletions, and passing those changes to other systems almost immediately.
It's how a warehouse, search index, or cache stays in sync with an operational database without heavy nightly copies. While CDC sounds like a niche data engineering trick, it's become the backbone of real-time analytics and event-driven systems.
Change data capture (CDC), explained at three levels
At home
Imagine a shared family calendar. Instead of re-copying the whole calendar every night, someone texts everyone the moment an event is added or moved. CDC does that for databases, sending only the changes as they happen.
At work
CDC keeps reports, search, and other systems up to date within seconds of a change in the main database. It also puts less load on the database than repeatedly copying entire tables.
At home
Databases already keep a log of every change so they can recover after a crash. Log-based CDC reads that log and turns each change into an event, like 'order 1234 status changed from pending to shipped.'
At work
Approaches include log-based CDC (most efficient and complete), trigger-based CDC (database triggers write changes to a table), and query-based polling using timestamps (simple but misses deletes). Events usually flow through Kafka or a cloud streaming service into warehouses, caches, or microservices.
At home
CDC must handle the first full copy of the data, called the snapshot, and then switch to streaming changes without missing or duplicating any. In my opinion, that handover is the hardest part to get right, and it's where homemade CDC scripts usually fail.
At work
Plan for schema changes in source tables, ordering guarantees per key, exactly-once or idempotent processing downstream, and log retention so connectors can catch up after outages. CDC streams also carry personal data, so apply filtering and masking. Coordinate with database owners, since log settings and replication slots affect source performance.
Real-world examples
At home
Lena built a side project that showed live inventory for a friend's bike shop. Using Debezium on the shop's PostgreSQL database, the website updated within seconds whenever a bike was sold in the store.
At work
Marcus is a data engineer at a Canadian online pharmacy. Nightly copies meant dashboards were always a day behind. Switching to CDC let the operations team see order delays in near real time and reroute shipments the same afternoon.
How it compares
| feature | Change data capture (CDC) | Batch extracts |
|---|---|---|
| Freshness | Seconds | Hours to a day |
| Load on source | Low, reads logs | High during copies |
| Captures deletes | Yes, with log-based CDC | Often not |
| Complexity | Higher | Lower |
CDC trades simplicity for fresh data and lighter database load.
For analysts
Before paying for real-time data, ask who will act on it within minutes. I'd use CDC where faster data changes a decision, such as fraud, inventory, or logistics, and keep simpler batches elsewhere.
History & origin
Databases have long kept transaction logs for crash recovery and replication, and commercial replication tools such as Oracle GoldenGate (acquired by Oracle in 2009) used them to copy changes between systems. LinkedIn's work on Databus and later Apache Kafka in the early 2010s showed how changes could flow as streams to many consumers. Red Hat launched the open-source Debezium project around 2016, which read logs from MySQL, PostgreSQL, and others and published changes to Kafka. Cloud services, including AWS Database Migration Service and Google Datastream, made log-based CDC a managed feature. In my view, CDC finally killed the 'nightly batch' as the default, and good riddance.