Apache Pinot
| Company type | Open source |
|---|---|
| Industry | Software, Data storage |
| Founded | 2013 |
| Founder | |
| Headquarters | United States |
| Products | Distributed OLAP data store |
| Website | pinot |
Apache Pinot is a Java-based, open-source, distributed data store optimized for OLAP workloads with minimal latency. This column-oriented system is tailored for rapid analytics involving aggregations on immutable datasets, supporting real-time data ingestion.[1][2] It is engineered to deliver consistent low-latency responses, maintain data freshness measured in seconds, and provide fault tolerance alongside scalability.[3] The project's name derives from its creators’ analogy of extracting meaningful insights from diverse datasets, akin to pressing Pinot grapes into wine.[4]
Originally created at LinkedIn to fulfill the demand for real-time analytics solutions, Pinot currently supports production workloads at organizations such as Uber,[5] Microsoft,[6] and Factual.
History
Pinot was initiated within LinkedIn in 2013 to power various analytics use cases aimed at both users and business operations. Its debut application involved overhauling LinkedIn’s feature that displays profile viewers in real-time. The software was released as open source under the Apache 2.0 license in June 2015 and subsequently contributed to the Apache Software Foundation in June 2019.
Architecture
Pinot incorporates Apache Helix for managing its cluster, embedding Helix agents into its components while utilizing Apache ZooKeeper to coordinate and preserve cluster state. Helix oversees all Pinot brokers and servers, orchestrating replica placement and partitioning across the distributed environment.
Query management
Incoming queries are handled by brokers that reference a segment-to-server routing map to distribute requests appropriately between offline and real-time server nodes.
Cluster management
Cluster orchestration relies on Apache Helix to manage partitioned and replicated resources, with cluster metadata and state stored in ZooKeeper.
Features
Sharing commonalities with OLAP systems like Apache Druid, Pinot is a columnar database that applies compression techniques including Run Length and Fixed-Bit Length. It distinguishes itself through support for modular indexing methods such as Sorted Index, Bitmap Index, Inverted Index, Star-Tree Index, and Range Index.
Pinot facilitates near real-time data ingestion from streaming platforms like Kafka and AWS Kinesis, alongside batch loading from sources including Hadoop, S3, Azure, and GCS. Its SQL-like query interface supports selection, aggregation, filtering, grouping, ordering, and distinct operations, aligning with typical OLAP and data warehousing functionalities.
See also
External links
- ↑ Lua error in Module:Citation/CS1/Configuration at line 2123: attempt to index field '?' (a nil value).
- ↑ Lua error in Module:Citation/CS1/Configuration at line 2123: attempt to index field '?' (a nil value).
- ↑ Lua error in Module:Citation/CS1/Configuration at line 2123: attempt to index field '?' (a nil value).
- ↑ Cite error: Invalid
<ref>tag; no text was provided for refs namedopen-sourcing-pinot - ↑ Lua error in Module:Citation/CS1/Configuration at line 2123: attempt to index field '?' (a nil value).
- ↑ Cite error: Invalid
<ref>tag; no text was provided for refs namedpinot-joins-apache-foundation