Apache Iceberg
Also known as: Iceberg, Iceberg table format
Apache Iceberg is an open table format: a specification for tracking which data files make up a table, along with its schema and history, so that files in cloud storage behave like reliable database tables.
Spark, Trino, Flink, Snowflake, and many other engines can read and write the same Iceberg tables. While a table format sounds like plumbing, Iceberg became the centre of a multi-billion-dollar fight over who controls enterprise data.
Apache Iceberg, explained at three levels
At home
Imagine a photo album where the pages are loose photos scattered in a drawer, plus an index card that says exactly which photos belong to the album, in what order. Iceberg is that index card for data files.
At work
Iceberg lets companies keep data in cheap cloud storage while still using it like a database. Different teams can use different tools on the same data without making copies.
At home
Iceberg keeps a tree of metadata files. Each change creates a new 'snapshot' of the table, so you can see what the data looked like last week (time travel) or undo a bad update. Columns can be renamed or added without rewriting data.
At work
Iceberg features include ACID transactions, hidden partitioning (queries don't need to know the partition layout), schema and partition evolution, and snapshot isolation. A catalogue, such as Apache Polaris, AWS Glue, or a REST catalogue, tracks the current metadata for each table.
At home
Iceberg updates tables by atomically swapping a pointer to a new metadata file, a simple trick that makes concurrent writes safe without a central database server. In my view, that design is elegant precisely because it relies on so little.
At work
Operational concerns include compaction of small files, expiring old snapshots to control storage, managing delete files from row-level updates, and choosing a catalogue that supports your engines and governance needs. Compare Iceberg with Delta Lake on ecosystem fit; interoperability layers are narrowing the gap, but they aren't seamless yet.
Real-world examples
At home
Carlos was building a portfolio project analyzing hockey statistics. Storing the data in Iceberg let him query it with both Spark and DuckDB, and roll back when a bad import duplicated a season.
At work
Lena is a data architect at a Toronto insurer whose analysts used Snowflake while data scientists used Spark. Moving shared tables to Iceberg let both groups query one copy of the claims data, cutting storage and ending sync jobs.
How it compares
| feature | Apache Iceberg | Delta Lake |
|---|---|---|
| Origin | Netflix, 2017 | Databricks, 2019 |
| Governance | Apache Software Foundation | Linux Foundation |
| Strongest ecosystem | Multi-vendor: Snowflake, AWS, Trino | Databricks and Spark |
| Partitioning | Hidden partitioning | Explicit partitions or clustering |
Both formats turn files into tables. The choice is mostly about which ecosystem you live in.
For analysts
Open table formats reduce lock-in, but the catalogue can bring it right back. I'd ask any vendor whether other engines can read and write your Iceberg tables through their catalogue, not just theirs.
History & origin
Ryan Blue and Dan Weeks created Iceberg at Netflix around 2017 to fix problems with Hive tables, such as slow file listings and unsafe concurrent writes. Netflix donated it to the Apache Software Foundation in 2018, and it became a top-level project in 2020. Blue and Weeks founded Tabular to commercialize it. In June 2024, during Snowflake's summit, Databricks announced it would acquire Tabular, for a price reported at well over US$1 billion, while Snowflake open-sourced its Polaris catalogue for Iceberg. AWS launched S3 Tables with built-in Iceberg support that December. In my opinion, when rival vendors all rush to support the same open format, customers have finally won a round.