Apache Flink
| Company type | Open source software |
|---|---|
| Industry | Stream processing, big data |
| Predecessor | Stratosphere |
| Founded | March 2014 |
| Founder | Volker Markl (project originator) |
Area served | Worldwide |
Key people | Apache Flink Community, 119 committers, 340+ contributors |
| Products | Stream processing engine, DataStream API, DataSet API, Table API, SQL interface |
| Owner | Apache Software Foundation |
| Parent | Apache Software Foundation |
| Website | flink |
Apache Flink is an open-source platform designed for unified stream processing and batch processing, maintained by the Apache Software Foundation. At its heart lies a distributed streaming dataflow engine implemented in Java and Scala. Flink is capable of running arbitrary Dataflow programming applications with support for Data parallelism and a pipelined execution model. Its execution environment natively accommodates both batch and stream processing workloads, including iterative algorithm execution.
Flink delivers a streaming engine characterized by high throughput and minimal latency, featuring event-time handling and robust state management. It guarantees fault tolerance along with exactly-once processing semantics. Developers can build applications using Java, Python, and SQL, which are automatically compiled and optimized into executable dataflow programs that operate efficiently on clusters or cloud infrastructures.
While Flink integrates seamlessly with numerous external data sources and sinks, it does not provide its own storage solution. Supported connectors encompass technologies such as Apache Doris, Amazon Kinesis, Apache Kafka, HDFS, Apache Cassandra, and ElasticSearch.
Development
The Apache Flink project is developed under the terms of the Apache License 2.0 by the Apache Flink Community within the umbrella of the Apache Software Foundation. The initiative benefits from the contributions of 119 committers alongside more than 340 additional contributors.
Overview
The dataflow programming model employed by Apache Flink facilitates event-at-a-time processing for both finite and infinite data streams. Programs in Flink are composed of streams—potentially unbounded sequences of data records—and transformations that process one or multiple streams to generate output streams.
Flink offers two primary APIs: the DataStream API, suitable for both bounded and unbounded streams, and the DataSet API, tailored to bounded datasets. Furthermore, it provides a Table API, which resembles SQL syntax for relational stream and batch processing and integrates seamlessly with the DataStream and DataSet APIs. The most abstracted language available is SQL itself, closely aligned with the Table API, enabling users to express programs as SQL queries.
Programming Model and Distributed Runtime
Flink applications run as streaming dataflows starting from one or more source inputs—such as message queues or file systems—and culminating in sinks like databases or file storage systems. These streams undergo various transformations arranged within a directed acyclic graph structure, supporting complex dataflow branching and merging.
Built-in connectors enable integration with widely used platforms including Apache Kafka, Amazon Kinesis, HDFS, and Apache Cassandra.
Operating as a distributed system across clusters, Flink supports deployment in standalone mode as well as on resource managers like YARN, Mesos, Docker-based setups, among others.
State: Fault Tolerance, Checkpoints, and Savepoints
Apache Flink incorporates a streamlined fault tolerance system built around distributed checkpoints. These checkpoints function as asynchronous, automatic snapshots capturing both the application's state and the position within the source stream. If a failure occurs, Flink applications with checkpointing activated restart from the most recent successful checkpoint, guaranteeing exactly-once processing semantics. This approach also enables seamless coordination with external systems, including transactional databases.
In addition to checkpoints, Flink provides savepoints—manually initiated snapshots—that permit users to halt and later resume applications from an identical state and stream offset. Savepoints are instrumental for upgrading Flink jobs or clusters without losing state information, and they allow restarting applications with altered parallelism levels to better handle varying workloads.
DataStream API
The DataStream API in Flink supports more than twenty types of transformations—including filters, aggregations, and window operations—applicable to both bounded and unbounded streams. This API is accessible in Java and Scala languages.
A common use case is a stateful streaming application that counts words from an ongoing input stream, grouping them into windows of five seconds:
<syntaxhighlight lang="scala"> import org.apache.flink.streaming.api.scala._ import org.apache.flink.streaming.api.windowing.time.Time
case class WordCount(word: String, count: Int)
object WindowWordCount {
def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val text = env.socketTextStream("localhost", 9999)
val counts = text.flatMap { _.toLowerCase.split("\\W+") filter { _.nonEmpty } }
.map { WordCount(_, 1) }
.keyBy("word")
.timeWindow(Time.seconds(5))
.sum("count")
counts.print
env.execute("Window Stream WordCount")
}
} </syntaxhighlight>
Apache Beam - Flink Runner
Apache Beam provides a unified programming framework for both batch and streaming data processing across various execution engines. Among the supported runners, the Apache Flink runner stands out as the most comprehensive in terms of features.
Data Artisans partnered closely with the Apache Flink and Beam communities to develop this runner, enhancing compatibility and broadening Flink’s presence within unified data processing ecosystems.
DataSet API
Flink's DataSet API facilitates transformations such as filtering, mapping, joining, and grouping on bounded datasets, offering over twenty transformation types. It supports Java, Scala, and an experimental Python interface. Conceptually akin to the DataStream API, the DataSet API was deprecated starting with Flink version 2.0.
Table API and SQL
The Table API in Flink offers a SQL-inspired language designed for relational processing of both streaming and batch data. This API is integrated within Flink’s Java and Scala DataSet and DataStream APIs. It operates on a relational Table abstraction derived from external sources or existing streams and datasets. Supported relational operations include selection, aggregation, and joins. Users can also query Tables using standard SQL syntax; both interfaces deliver equivalent capabilities that can be combined within a single program. The logical query plans are optimized via Apache Calcite before being translated into DataSet or DataStream execution plans.
Flink Forward
Flink Forward is an annual gathering dedicated to Apache Flink. It debuted in 2015 in Berlin, attracting over 250 participants and featuring a combination of technical talks and practical workshops. The following year, attendance increased to 350, with a broader array of sessions and presenters.
In 2017, the conference extended its reach to San Francisco, placing greater emphasis on enterprise applications, system architecture, ecosystem integrations, and upcoming platform innovations through keynote speeches, industry discussions, and training sessions.
Due to the COVID-19 pandemic, the spring 2020 edition transitioned to a virtual event format, offering live keynotes and presentations covering Flink’s internal workings, stream processing techniques, real-time analytics, and practical use cases.
Marking its 10th anniversary in 2024 back in Berlin, Flink Forward spotlighted plans for Flink 2.0, including the discontinuation of Java 8 support and the rollout of a new state backend. The conference also introduced Flink CDC for no-code workflows authored in YAML and highlighted Flink’s integration with OpenLineage.
History
The origins trace back to 2010 with the research initiative "Stratosphere: Information Management on the Cloud," spearheaded by Volker Markl and funded by the German Research Foundation (DFG). This project was a joint effort among Technische Universität Berlin, Humboldt-Universität zu Berlin, and the Hasso-Plattner-Institut Potsdam. Apache Flink emerged from a fork of Stratosphere's distributed execution engine and entered the Apache Incubator in March 2014. By December of that year, it had graduated to an Apache top-level project, reflecting its readiness for enterprise adoption.
} Highlights from Recent Releases- March 2025: Apache Flink version 2.0 (v2.0.0)
- August 2024: Apache Flink Long-Term Support release (v1.20.1)
- March 2024: Apache Flink version 1.19 (v1.19.x)
- October 2023: Apache Flink version 1.18 (v1.18.x)
See also
External links
| Version | Initial Release Date | Latest Version | Latest Release Date |
|---|---|---|---|
| 2015-06-24 | 0.9.1 | 2015-09-01 | |
| 2015-11-16 | 0.10.2 | 2016-02-11 | |
| 2016-03-08 | 1.0.3 | 2016-05-11 | |
| 2016-08-08 | 1.1.5 | 2017-03-22 | |
| 2017-02-06 | 1.2.1 | 2017-04-26 | |
| 2017-06-01 | 1.3.3 | 2018-03-15 | |
| 2017-12-12 | 1.4.2 | 2018-03-08 | |
| 2018-05-25 | 1.5.6 | 2018-12-26 | |
| 2018-08-08 | 1.6.3 | 2018-12-22 | |
| 2018-11-30 | 1.7.2 | 2019-02-15 | |
| 2019-04-09 | 1.8.3 | 2019-12-11 | |
| 2019-08-22 | 1.9.2 | 2020-01-30 | |
| 2020-02-11 | 1.10.3 | 2021-01-29 | |
| 2020-07-06 | 1.11.6 | 2021-12-16 | |
| 2020-12-10 | 1.12.7 | 2021-12–16 | |
| 2021–05–03 | 1.13.6 | 2022–02–18 | |
| 2021–09–29 | 1.14.6 | 2022–09–28 | |
| 2022–05–05 | 1.15.4 | 2023–03–15 | |
| 2022–10–28 | 1.16.3 | 2023–11–29 | |
| 2023–03–23 | 1.17.2 | 2023–11–29 | |
| 2023–10–24 | 1.18.1 | 2024–01–19 | |
| 2024–03–18 | 1.19.2 | 2025–02–12 | |
| 2024–08–02 | 1.20.1 | 2025–02–12 | |
| 2025–03–19 | 2.0.0 | 2025–03–19 |