Apache Spark
Also known as: PySpark, Spark SQL, Structured Streaming, MLlib, SparkContext, Databricks Spark
Apache Spark is an open-source distributed data processing engine.
It processes large datasets across clusters of computers simultaneously, supporting batch processing, real-time streaming, SQL queries, and machine learning. Spark is the processing engine inside Databricks and many data lake platforms.
Apache Spark, explained at three levels
At home
Spark processes huge amounts of data by splitting the work across many computers simultaneously. Instead of processing a billion records one at a time on one machine, Spark divides them into chunks, processes each chunk on a different server, and combines the results, reducing hours to minutes.
At work
Data engineering teams use Spark to build pipelines that clean, transform, and aggregate large datasets. A job that would take hours on a single machine runs in minutes on a Spark cluster. Databricks, the most popular Spark platform, makes running Spark jobs simple without managing cluster infrastructure.
At home
Spark's core concepts: RDD (Resilient Distributed Dataset, the original API), DataFrame (structured tabular data with schema, similar to a database table), Dataset (type-safe DataFrame in Scala/Java). The Spark engine optimizes query plans using Catalyst, then executes using Tungsten (off-heap memory, SIMD optimization). Spark supports four libraries: Spark SQL (structured data), Spark Streaming/Structured Streaming (real-time), MLlib (machine learning), and GraphX (graph processing).
At work
PySpark (Python API) is the most widely used Spark interface. A Spark job runs as a Driver program (the main program) that distributes work to Executors (workers on cluster nodes). Lazy evaluation: transformations (filter, map, join) are recorded as a plan; actions (count, write, collect) trigger actual execution. This enables the Catalyst optimizer to reorder operations and prune unnecessary work.
At home
Spark's execution model uses stages (groups of tasks that can run in parallel), separated by shuffle operations (redistributing data across all nodes by key). Shuffles are expensive: they require network transfer of data between executors. Optimizing shuffle (using broadcast joins for small tables, partitioning data by join keys) is the primary Spark performance tuning task.
At work
Structured Streaming treats real-time data as an unbounded table, processing micro-batches (100ms-1s intervals) or using continuous processing mode. Watermarking handles late-arriving data by defining how long to wait before computing a window. Adaptive Query Execution (Spark 3.0) dynamically adjusts query plans based on runtime statistics, handling data skew and repartitioning automatically. Photon (Databricks' C++ query engine) provides 2-8x performance improvements over standard Spark for SQL workloads.
Real-world examples
At home
Aisha's data science project analyzes 5 years of GPS data from 10 million rides. On her laptop, the analysis would take 8 hours. She submits the same PySpark code to a Databricks cluster with 20 nodes: it finishes in 12 minutes.
At work
Jerome's company runs daily Spark ETL jobs on Databricks. Raw event data in S3 (10 TB per day) is read, cleaned, joined with customer dimension tables, and written to Delta Lake partitioned tables. The pipeline runs in 45 minutes using a 50-node cluster that terminates after the job: zero idle cost.
Common confusions
- Spark replaces your database — Spark is a processing engine, not a storage system. It reads from and writes to storage (S3, Delta Lake, HDFS, databases) but doesn't store data itself. You use Spark to transform and analyze data; you store results in a database, data lake, or data warehouse.
- You need to know Scala to use Spark — PySpark (Python) is the most popular Spark interface and is first-class in Databricks. Spark also has Java, R, and SQL interfaces. Scala provides the best performance for low-level operations but isn't required for most data engineering and science work.
For analysts
Spark job performance and cost are directly linked: a poorly optimized Spark job might run for 4 hours on a 100-node cluster; an optimized one runs in 20 minutes on 20 nodes. BAs involved in data platform cost optimization should understand that Spark tuning is a real engineering discipline with significant cost impact.
History & origin
Spark was created at UC Berkeley's AMPLab in 2009 and open-sourced in 2010. It was designed as a faster alternative to Hadoop MapReduce, using in-memory computation instead of writing intermediate results to disk. Spark became a top-level Apache project in 2014 and rapidly displaced Hadoop MapReduce for big data processing. The same UC Berkeley team founded Databricks (2013) to commercialize Spark. Spark is now the de facto engine for large-scale data processing in enterprise data platforms.