Apache Parquet

Apache Parquet
Company type	Open-source software project
Industry	Data storage, Big data
Founded	July 2013; 13 years ago
Founder	Twitter and Cloudera
Headquarters	Global (Apache Software Foundation project)
Products	Column-oriented data storage format
Parent	Apache Software Foundation
Website	parquet.apache.org

Apache Parquet is a free and open-source column-oriented storage format for data, integrated within the Apache Hadoop framework. Similar to RCFile and ORC, Parquet is compatible with most Hadoop ecosystem data processing tools, providing advanced data compression and encoding techniques to enhance efficiency when handling intricate, large-volume datasets.

History

The creation of Apache Parquet as an open-source project was a joint effort between Twitter and Cloudera. It was developed to surpass the capabilities of Trevni, a columnar storage format crafted by Doug Cutting, the founder of Hadoop. The term 'parquet'—meaning "small compartment"—was selected to symbolize the foundational structure of a database with a unique, organized pattern. Apache Parquet version 1.0 was released in July 2013, and on April 27, 2015, it achieved status as a top-level project under the Apache Software Foundation (ASF).

Features

Apache Parquet employs a record-shredding and reassembly process that supports sophisticated data structures, enabling efficient storage. Data values are stored sequentially by column, which provides several advantages:

Column-specific compression minimizes disk space usage
Encoding and compression techniques can be customized based on each column’s data type
Queries focusing on particular columns avoid scanning entire rows, resulting in improved query speed

Built upon the Apache Thrift framework, Parquet offers support for various programming languages including C++, Java, Python, and PHP.

By August 2015, Parquet had been integrated into key big data platforms such as Apache Hive, Apache Drill, Apache Impala, Apache Crunch, Apache Pig, Cascading, Presto, and Apache Spark. Additionally, it is utilized by the Python library pandas for data manipulation and analysis.

Compression and encoding

Parquet applies compression on a per-column basis, allowing different encoding strategies for text versus numeric data, and facilitating the adoption of new encoding techniques over time.

The compression algorithms supported include Snappy, gzip, LZO, brotli, zstd, and LZ4.

Dictionary encoding

Parquet automatically uses dictionary encoding for datasets with a limited number of distinct values (fewer than 10⁵), which significantly enhances compression ratios and processing speed.

Bit packing

Rather than allocating fixed 32 or 64 bits per integer, Parquet compresses multiple small integers into shared storage space, increasing storage efficiency.

Run-length encoding (RLE)

To efficiently represent sequences of repeated values, Parquet employs run-length encoding, storing the value once alongside its repetition count.

Parquet combines bit packing with RLE in a dynamic manner, switching between these methods to achieve optimal compression. This combined technique works well with dictionary encoding to further improve performance.

Cloud Storage and Data Lakes

Parquet is widely adopted as a file format within cloud-based data lake infrastructures. Leading cloud storage services such as Amazon S3, Azure Data Lake Storage, and Google Cloud Storage frequently utilize Parquet due to its efficient column-oriented storage and rapid data access capabilities. Data lakehouse solutions like Apache Iceberg, Delta Lake, and Apache Hudi implement metadata layers on Parquet files to support features including schema evolution, time-travel querying, and ACID-compliant transaction management. Within these architectures, Parquet functions as the immutable data storage foundation, while table formats manage version control and transactional consistency.

Comparison

Within the Hadoop ecosystem, Apache Parquet competes with other columnar storage formats like RCFile and Optimized Row Columnar (ORC), all designed to deliver enhanced compression and improved read efficiency though often at the cost of slower write speeds. Parquet also offers limited schema evolution capabilities, enabling adjustments such as adding columns or merging compatible schemas.

Apache Arrow acts as an in-memory complement to on-disk columnar formats like Parquet and ORC. Both projects provide interoperable libraries that facilitate seamless reading and writing between Arrow and Parquet data representations.

Implementations

Key implementations of the Parquet format include: