Data lake
Also known as: datalake, data lakes
A data lake is a large, central storage system that holds raw data in its original format, structured tables, logs, documents, images, and sensor readings, until someone needs to analyze it.
Data lakes are usually built on low-cost cloud object storage such as Amazon S3 or Azure Data Lake Storage. They support data science, machine learning, and big data analytics. While data lakes are flexible and cheap, poor organization can turn them into a 'data swamp' where nobody can find or trust anything.
Data lake, explained at three levels
At home
A data lake is like a huge storage unit where you keep everything, boxes, photos and papers, without sorting it first. Space is cheap, so you keep it all in its original form. When you finally need something, you search through and organize just that part. A data lake does the same with raw files, applying structure only when someone analyzes them.
At work
A data lake is a large, central store that keeps an organization's raw data in its original format, such as logs, spreadsheets, documents and sensor readings, usually on low-cost cloud storage. Data is organized when someone analyzes it, not when it is saved. Businesses use data lakes to keep large amounts of varied data affordably for reporting, data science and AI projects.
At home
A data lake stores data first and decides how to use it later. Instead of cleaning everything into fixed tables before saving, as a traditional data warehouse does, the lake accepts raw files of any kind: spreadsheets, logs, images and sensor readings, in their original format, on cheap cloud storage. When someone needs an answer, query tools read the relevant files and apply structure at that moment, an approach called schema on read.
At work
Lakes typically sit on object storage such as Amazon S3, Azure Data Lake Storage or Google Cloud Storage, holding files in formats such as Parquet, a compressed columnar format that speeds queries. A catalogue records what each dataset contains, and engines such as Spark, Trino or Athena query it. Table formats such as Delta Lake, Apache Iceberg and Hudi add transactions and versioning, creating the lakehouse pattern that blends lake and warehouse features.
At home
Because storage is cheap, companies often keep everything, including old data about customers who left years ago. That data can be breached, misused or combined in ways you never expected. Canadian privacy law expects organizations to keep personal information only as long as needed for its purpose. In my view, asking businesses to delete your data when you leave is worthwhile, since data that no longer exists cannot leak.
At work
Without governance, data lakes become data swamps: undocumented, duplicated and untrusted. Implement catalogues, ownership, classification, lineage and retention policies from the start, with zones for raw, cleaned and curated data. Restrict access to sensitive zones with fine-grained permissions, encrypt storage and log queries. Monitor storage and compute costs as volumes grow, and apply retention and deletion workflows that satisfy privacy obligations, including deleting personal data from historical files and table versions.
Real-world examples
At home
Fatima's fitness app collects raw data from her watch, heart rate, steps, and sleep, and keeps it so new features can analyze it later.
At work
Marcus leads analytics at a Calgary energy company. He built a data lake for sensor readings from pipelines, with a catalogue so analysts can find trusted datasets.
How it compares
| feature | Data lake | Data warehouse |
|---|---|---|
| Data format | Raw, any format | Cleaned, structured |
| Structure applied | When read | When loaded |
| Cost | Lower storage cost | Higher |
| Main users | Data scientists, engineers | Business analysts |
Lakes store everything raw. Warehouses store curated data for reporting.
For analysts
Before building a data lake, I'd identify the first business questions it should answer. Lakes built 'just in case' often become swamps.
History & origin
James Dixon, then chief technology officer of the analytics company Pentaho, coined the term 'data lake' in 2010, contrasting it with data marts, which he likened to bottled water. The Hadoop software framework, which emerged around 2006, made storing huge amounts of raw data affordable, and cloud object storage made it even cheaper in the 2010s. Many early data lakes became disorganized, leading to the term 'data swamp.' Around 2020, 'lakehouse' designs from companies such as Databricks combined data lake storage with data warehouse features, using open table formats such as Delta Lake and Apache Iceberg. In my view, a data lake without governance is just an expensive junk drawer.