Data lake
Also known as: datalake, data lakes
A data lake is a large, central storage system that holds raw data in its original format, structured tables, logs, documents, images, and sensor readings, until someone needs to analyze it.
Data lakes are usually built on low-cost cloud object storage such as Amazon S3 or Azure Data Lake Storage. They support data science, machine learning, and big data analytics. While data lakes are flexible and cheap, poor organization can turn them into a 'data swamp' where nobody can find or trust anything.
Data lake, explained at three levels
At home
A data lake is like a huge storage unit where you keep everything, boxes, photos, papers, without sorting it first. You organize what you need when you need it.
At work
Data lakes let organizations keep all kinds of data affordably for future analysis and AI projects.
At home
Your phone's photo library is a tiny version of a data lake: lots of raw files, and search tools help you find what you need later.
At work
Data lakes store data in formats such as Parquet and CSV on object storage, with catalogues, access controls, and query engines added on top.
At home
Companies sometimes keep data far longer than needed. In my opinion, asking businesses to delete your data when you leave is worthwhile.
At work
Implement data catalogues, ownership, classification, lineage, and retention policies from the start. Restrict access to sensitive zones, encrypt storage, and monitor costs as data volumes grow.
Real-world examples
At home
Fatima's fitness app collects raw data from her watch, heart rate, steps, and sleep, and keeps it so new features can analyze it later.
At work
Marcus leads analytics at a Calgary energy company. He built a data lake for sensor readings from pipelines, with a catalogue so analysts can find trusted datasets.
How it compares
| feature | Data lake | Data warehouse |
|---|---|---|
| Data format | Raw, any format | Cleaned, structured |
| Structure applied | When read | When loaded |
| Cost | Lower storage cost | Higher |
| Main users | Data scientists, engineers | Business analysts |
Lakes store everything raw. Warehouses store curated data for reporting.
For analysts
Before building a data lake, I'd identify the first business questions it should answer. Lakes built 'just in case' often become swamps.
History & origin
James Dixon, then chief technology officer of the analytics company Pentaho, coined the term 'data lake' in 2010, contrasting it with data marts, which he likened to bottled water. The Hadoop software framework, which emerged around 2006, made storing huge amounts of raw data affordable, and cloud object storage made it even cheaper in the 2010s. Many early data lakes became disorganized, leading to the term 'data swamp.' Around 2020, 'lakehouse' designs from companies such as Databricks combined data lake storage with data warehouse features, using open table formats such as Delta Lake and Apache Iceberg. In my view, a data lake without governance is just an expensive junk drawer.