Apache Kafka

Apache Kafka

Company type	Open source software
Industry	Software, Stream processing
Founded	2011 (2011)
Founder	Jay Kreps, Neha Narkhede, Jun Rao
Products	Distributed event streaming platform, Kafka Connect, Kafka Streams
Parent	Apache Software Foundation

Apache Kafka serves as a distributed event store and stream-processing platform. This open-source technology, developed under the Apache Software Foundation, is implemented in both Java and Scala. Kafka delivers a comprehensive, high-throughput, and low-latency solution for processing real-time data streams. It integrates with external systems through Kafka Connect and supports stream processing applications via Kafka Streams libraries. Kafka's communication relies on a binary protocol over TCP, designed for efficiency by grouping messages into "message sets" to minimize network overhead. This architecture facilitates larger network packets, sequential disk writes, and contiguous memory usage, transforming sporadic random writes into streamlined linear disk operations.

History

Initially created at LinkedIn, Kafka was made open source in early 2011. The platform was co-founded by Jay Kreps, Neha Narkhede, and Jun Rao. It officially graduated from the Apache Incubator on October 23, 2012. The name "Kafka" was chosen by Jay Kreps, inspired by author Franz Kafka, reflecting the system’s focus on optimized writing processes and Kreps’ admiration for the writer’s work.

Operation

Apache Kafka functions as a distributed log-centric messaging system that ensures message ordering within individual partitions rather than across an entire topic. Unlike traditional queue systems, Kafka stores messages in a durable, append-only log format, allowing multiple consumers to read independently at various offsets. It uses manual offset control, giving consumers the ability to manage retries and error handling themselves. In cases where a consumer cannot process a message, it may postpone committing the offset, pausing progress on that partition while other partitions continue unaffected. This partition-based approach supports fault isolation and parallelism while preserving order within each partition according to consumer processing.

In 2025, Apache Kafka introduced "Queues for Kafka," which added share groups as an alternative to consumer groups. This innovation provides queue-like behavior where consumers collaboratively handle records from shared partitions with individual message acknowledgments and delivery monitoring. Unlike traditional consumer groups that assign exclusive partitions per consumer, share groups permit more consumers than partitions, making them well-suited for work-queue scenarios while maintaining Kafka’s durability and scalability. This enhancement addresses the frequent issue of "over-partitioning" encountered by many Kafka users.

Connect API

Kafka Connect is a framework designed to facilitate data transfer between Kafka and external systems. Launched in version 0.9.0.0, it internally leverages the Producer and Consumer APIs. The framework operates "connectors" that contain the logic for data integration tasks. The Connect API specifies the interface required to develop custom connectors. While many open source and commercial connectors are available, Apache Kafka itself does not supply connectors that are production-ready out of the box.

Streams API

Introduced in version 0.10.0.0, Kafka Streams is a Java library designed for stream processing that enables the creation of scalable, resilient, and stateful applications. Its primary interface is a stream-processing DSL that provides a variety of operators such as filtering, mapping, grouping, windowing, aggregation, joining streams, and managing tables. Additionally, the Processor API offers the capability to build custom operators and can be integrated alongside the DSL. For handling state, Kafka Streams employs RocksDB to store operator state locally on disk, which allows for managing data sets exceeding available memory. Changes to these local stores are simultaneously recorded to Kafka topics to ensure fault tolerance, facilitating state restoration by replaying these topics back into RocksDB.

See also

• RabbitMQ
• Redis
• NATS
• Apache Flink
• Apache Samza
• Apache Spark Streaming
• Data Distribution Service
• Enterprise Integration Patterns
• Enterprise messaging system
• Streaming analytics
• Event-driven SOA
• Hortonworks DataFlow
• Message-oriented middleware
• Service-oriented architecture