Data deduplication
Also known as: deduplication, dedupe, dedup
Data deduplication is a technique that finds repeated chunks of data and stores them only once, replacing duplicates with small pointers to the single copy.
It's widely used in backup systems, where many computers contain the same operating system files and daily backups repeat most of yesterday's data, often cutting storage needs by large factors. While deduplication saves money, it adds processing work and means one corrupted chunk could affect many files, so integrity checks matter.
Data deduplication, explained at three levels
At home
Deduplication is like a library keeping one copy of a popular book and giving everyone a note pointing to it, instead of buying a hundred copies.
At work
Deduplication reduces backup storage costs by avoiding saving the same data again and again.
At home
Duplicate photo finder apps do a simple version of deduplication, finding identical files so you can delete extras.
At work
Deduplication can happen at the source before sending data, or at the target storage. Ratios depend on data type; encrypted or compressed data deduplicates poorly.
At home
Deleting duplicate photos frees space, but make sure you keep the highest-quality version. In my opinion, review before bulk-deleting.
At work
Plan restore performance, since rehydrating deduplicated data can be slower, verify integrity regularly, consider ransomware's effect on dedup ratios, and size storage on realistic ratios, not vendor best cases.
Real-world examples
At home
Fatima ran a duplicate finder on her laptop and discovered 40 GB of repeated photos from years of phone imports.
At work
Jerome's Regina MSP backs up 50 client PCs with deduplication. Since most share identical Windows and Office files, total backup storage stays small.
How it compares
| feature | Data deduplication | Compression (making files smaller) |
|---|---|---|
| How it saves space | Removes repeated chunks | Encodes data more efficiently |
| Works best on | Many similar files | Text and uncompressed data |
| Common in | Backup systems | Files, archives, networks |
| Used together | Often | Often |
Deduplication removes repeats. Compression shrinks what's left.
For analysts
When comparing backup vendors, I'd ask for deduplication ratios on data like ours, not marketing averages. It changes storage cost estimates a lot.
History & origin
Simple single-instance storage, keeping one copy of identical files or email attachments, appeared in systems such as Microsoft Exchange in the 1990s. Data Domain, founded in 2001, popularized block-level deduplication for backups and was acquired by EMC in 2009 for about $2.1 billion US. Deduplication spread into backup software, storage arrays, and operating systems, including Windows Server 2012. Cloud backup services rely on deduplication to keep costs low. In my view, deduplication is a quiet reason backups became affordable for small businesses.