Apache Kafka
Also known as: Kafka, Confluent Kafka, Amazon MSK, Kafka Streams, event streaming
Apache Kafka is an open-source distributed event streaming platform.
Producers publish messages to topics; consumers subscribe and process them -- at any scale. Kafka can handle millions of events per second with low latency, making it the backbone for real-time analytics, event-driven architectures, and data pipeline integration.
Apache Kafka, explained at three levels
At home
Kafka is a high-speed message delivery system. One service (a producer) drops messages into Kafka; another service (a consumer) picks them up and processes them. Kafka holds the messages even if the consumer is temporarily offline, ensuring no messages are lost.
At work
Companies use Kafka to connect systems that produce data (apps, sensors, databases) with systems that consume it (analytics, dashboards, search indexes). Instead of each producer directly notifying each consumer, they all connect through Kafka -- one hub that handles routing, buffering, and redelivery.
At home
Kafka's key concepts: Topics (named message streams), Partitions (parallelism unit within a topic), Producers (write messages to topics), Consumers (read from topics, track position with 'offsets'), Consumer Groups (multiple consumers sharing a topic's partitions for parallel processing), Brokers (Kafka servers that store messages), and Retention (messages stored for a configurable period, not just until consumed).
At work
Common Kafka patterns: Event sourcing (every state change is an event; current state is computed from event log), CDC (Change Data Capture -- capturing database changes as Kafka events via Debezium), stream processing (Kafka Streams or Apache Flink processes events in motion), and data lake ingestion (streaming events to S3 or Snowflake). Confluent Schema Registry enforces Avro/Protobuf schemas for data contracts between producers and consumers.
At home
Kafka's durability model: messages are appended to partitioned, replicated logs on broker disk. Replication factor (typically 3) ensures durability if brokers fail. The leader broker for a partition handles reads and writes; followers replicate. ISR (In-Sync Replicas) are followers caught up to the leader -- only ISR members can become leaders on failure. Message ordering is guaranteed within a partition; across partitions, ordering requires care.
At work
Kafka's exactly-once semantics (EOS) use idempotent producers, transactional APIs, and atomic read-process-write in Kafka Streams. KRaft (Kafka Raft) replaces ZooKeeper for metadata management, running Raft consensus in Kafka controllers -- simplifying architecture and enabling single-node Kafka for development. Tiered Storage (in progress in the community) allows Kafka to offload older log segments to object storage (S3), decoupling retention period from broker disk capacity.
Real-world examples
At home
Marcus's e-commerce site sends an 'order placed' event to a Kafka topic when a customer checks out. Three consumers read this topic independently: one sends a confirmation email, one updates inventory, one triggers shipping. Each processes at its own speed without affecting the others.
At work
Fatima's company streams 50 million user events per day to Kafka. A Kafka Streams application computes real-time session metrics; Debezium CDC replicates database changes to Kafka for downstream consumers; and a Kafka connector ships all events to Snowflake for next-day analytics -- all connected through Kafka.
Common confusions
- Kafka is just a message queue like RabbitMQ — Kafka retains messages for a configurable period (days to forever) and allows any consumer to replay from any point. Traditional queues delete messages when consumed. Kafka's log-based model enables replay, audit trails, and multiple independent consumers reading the same data.
- Kafka is only for big companies — Kafka is available as a managed service (Confluent Cloud, Amazon MSK, Azure Event Hubs Kafka-compatible) with low-cost entry points. Startups use Kafka for event-driven architectures long before they reach LinkedIn's scale. A single-node Kafka setup is sufficient for modest workloads.
For analysts
Kafka is often the upstream source for real-time dashboards and data pipelines. BAs working with data engineering teams should understand that Kafka topics are where business events originate -- purchase events, login events, sensor readings -- and that replay capability allows reprocessing historical events for new analytics.
History & origin
Kafka was created at LinkedIn by Jay Kreps, Neha Narkhede, and Jun Rao in 2010 to solve the real-time activity feed problem at massive scale. LinkedIn open-sourced it in 2011 and donated it to Apache in 2012. Confluent (the commercial company behind Kafka) was founded in 2014 by the LinkedIn creators. Kafka has become the de facto standard for event streaming, deployed at Uber, Netflix, Airbnb, Twitter, and thousands of other companies. KRaft mode (2022) removed Kafka's ZooKeeper dependency, simplifying operations.