Apache Iceberg

Apache Iceberg

Company type	Open-source software project
Industry	Big data, Data management, Analytics
Founded	2017 (2017)
Founder	Ryan Blue, Dan Weeks
Headquarters	United States
Area served	Worldwide
Products	Table format for analytic data lakes
Owner	Apache Software Foundation
Website	iceberg.apache.org

Apache Iceberg represents a robust open-source software format designed for managing extensive analytic tables. It facilitates the use of SQL tables tailored for big data scenarios and supports concurrent, secure access by multiple processing engines such as Spark, Trino, Flink, Presto, Hive, Impala, StarRocks, Doris, and Pig. The project is distributed under the terms of the Apache License.

By tackling the performance bottlenecks and usability limitations encountered with Apache Hive tables in complex data lake setups, Iceberg offers significant improvements. Prominent organizations backing Apache Iceberg include Buster, CelerData, Cloudera, Crunchy Data, Dremio, IBM watsonx.data, IOMETE, Snowflake, Starburst, Tabular, AWS, and Google Cloud.

History

Iceberg originated at Netflix through the efforts of Ryan Blue and Dan Weeks to overcome critical shortcomings of Apache Hive within Netflix’s technology stack. Hive’s inability to ensure data integrity and provide reliable atomic transactions discouraged many teams from performing modifications on Hive tables due to risk of errors. Ryan Blue's development goals for Iceberg were threefold:

1. Guarantee data accuracy with full support for ACID transactional operations.
2. Boost efficiency by implementing fine-grained file-level manipulations that optimize write processes.
3. Streamline table management and routine maintenance tasks.

The project commenced in 2017 and was released as an open-source initiative under the auspices of the Apache Software Foundation in November 2018. It achieved top-level Apache project status by May 2020.

Today, Iceberg is utilized by leading enterprises such as Airbnb, Apple, Expedia, LinkedIn, Adobe, and Lyft, among others.

Technical details

Apache Iceberg separates table metadata from the underlying storage layer by maintaining files that track snapshots, schema definitions, partition configurations, and the locations of data files, facilitating efficient and atomic operations on tables.

Data within Iceberg is structured into snapshots that capture the table's state at distinct moments in time. This design supports ACID-compliant transactions, including features like snapshot isolation, concurrent write operations, and rollback capabilities. The snapshot metadata is organized as a hierarchy of manifest and metadata files stored within the file system.

For storing data, Iceberg relies on the Apache Parquet format, chosen for its highly efficient columnar layout tailored for analytical workloads. Parquet files contain compressed column-based rows, which help reduce storage requirements and enhance read performance through techniques such as predicate pushdown and column pruning. Manifest files maintain references to these Parquet files, enabling rapid retrieval of pertinent data during query execution.

The metadata structure comprises a top-level table metadata file (metadata.json) that holds information on the schema, partition specifications, lists of snapshots, and pointers to the current root snapshot. Each snapshot connects to a manifest list formatted in Avro, which enumerates individual manifest files. These manifest files index data files with details like row counts, partition values, and column-level statistics. This segmented approach to metadata allows entire manifests to be excluded from queries based on partition pruning, avoiding the need for unwieldy monolithic metadata files. Iceberg preserves historical snapshots and manifests to support time travel functionality until they expire and improves query planning by selectively reading only necessary manifests rather than scanning all data files or directories—significantly boosting efficiency when handling large-scale tables.

See also

• List of Apache Software Foundation projects