Apache Parquet
Also known as: Parquet, Parquet file format
Apache Parquet is an open-source file format that stores data by column instead of by row.
That makes analytic queries, which usually need only a few columns from huge tables, much faster and cheaper, and it compresses data far better than formats like CSV. While most people never see a Parquet file directly, it quietly underpins modern data lakes, lakehouses, and many AI datasets.
Apache Parquet, explained at three levels
At home
Think of a phone book printed one column per page: all the names on one page, all the addresses on the next. If you only need addresses, you read one page instead of the whole book. Apache Parquet stores data that way, column by column, so a program asking for one piece of information, like everyone's province, can skip everything else.
At work
Apache Parquet is an open file format that stores tables column by column rather than row by row. Analytics questions usually need a few columns from huge tables, so Parquet lets tools read only those columns, and similar values stored together compress very well. Businesses care because switching from CSV to Parquet typically shrinks files and speeds up reports while lowering cloud storage and query costs.
At home
Storing a column together means its values look alike: many repeated province names, or dates close together. Parquet exploits that with encodings, such as dictionary encoding, which stores 'Ontario' once and then small numbers pointing to it, before compressing the result. Each file also records its column types and summary statistics, like minimum and maximum values, so tools know what's inside without reading everything.
At work
A Parquet file is split into row groups, each holding a chunk of rows; within each, every column is stored as a column chunk divided into pages. A footer records the schema and statistics for each chunk. Query engines read the footer first, skip row groups whose minimum and maximum values can't match a filter, called predicate pushdown, and read only the needed columns. It supports nested data and works with nearly every analytics tool.
At home
Parquet files are written once and never edited in place. Changing a single row means rewriting a whole file, because the compressed column chunks can't be patched. That makes Parquet superb for reading and awkward for updates. In my opinion, that's exactly why table formats like Iceberg exist: they manage many immutable Parquet files and track replacements, so tables seem updatable even though no file ever changes.
At work
Performance depends on layout choices: row group size, often 128 MB to 1 GB; the compression codec, with Snappy favouring speed and Zstandard favouring size; and sorting or clustering on common filter columns so statistics skip more data. Bloom filters and page indexes add finer skipping. Avoid many tiny files, which multiply metadata reads. Writers differ in encodings and newer logical types, so test compatibility across the tools that will read your files.
Real-world examples
At home
Marcus downloaded a public dataset of Canadian weather records as a 4 GB CSV file. Converting it to Parquet shrank it to under 600 MB, and his analysis notebook ran in seconds instead of minutes.
At work
Nadia manages a data platform at a streaming service. Switching daily event exports from JSON to Parquet cut cloud storage costs by more than half, and the analysts' most common dashboard queries ran several times faster.
How it compares
| feature | Apache Parquet | CSV |
|---|---|---|
| Storage layout | Columnar | Row-based text |
| Compression | Strong, built in | None unless zipped |
| Data types | Stored in the file | Everything is text |
| Human readable | No, needs a tool | Yes, open in any editor |
CSV is for sharing. Parquet is for analysis at scale.
For analysts
If your team exchanges large data extracts as CSV, switching to Parquet is one of the cheapest wins available. I'd keep CSV only for small files that people genuinely need to open in a spreadsheet.
History & origin
Parquet was created in 2013 by engineers at Twitter and Cloudera, inspired by Google's 2010 Dremel paper on columnar storage of nested data. It became a top-level Apache project in 2015 and quickly became the default format for Hadoop and Spark workloads. The related Apache Arrow project, launched in 2016, standardized the in-memory columnar format that pairs naturally with Parquet. Today, Iceberg, Delta Lake, and Hudi tables store their data mostly in Parquet files, and services from Snowflake to BigQuery read it natively. In my view, Parquet is one of the most successful open standards in data, precisely because nobody had to market it.