Apache Kylin
| Company type | Open source software |
|---|---|
| Industry | Data analytics, Business intelligence |
| Founded | 2013 |
| Founder | eBay |
| Headquarters | Shanghai , China |
| Products | Distributed analytics platform, OLAP engine |
| Parent | Apache Software Foundation |
| Website | kylin |
Apache Kylin is a open source distributed analytics platform that offers a SQL interface alongside multi-dimensional analysis (OLAP) capabilities on top of Hadoop and Alluxio. It is designed to handle and analyze extremely large-scale data sets.
Initially created by eBay, Apache Kylin has since become a project under the Apache Software Foundation.
History
The development of Kylin started in 2013 within eBay's research and development team based in Shanghai. In October 2014, version 0.6 was released as open source on GitHub under the name "KylinOLAP."
By November 2014, Kylin joined the Apache Software Foundation's incubator program and was promoted to a Top Level Project by December 2015.
In March 2016, the original developers established Kyligence, Inc., which provides a commercial analytics solution built on Apache Kylin, supporting both cloud and on-premises data environments.
Architecture
Apache Kylin is constructed atop technologies such as Apache Hadoop, Hive, HBase, Parquet, Calcite, and Spark. This foundation allows it to efficiently scale when processing very large volumes of data.
The main components of Kylin include:
- REST Server: Responsible for managing user interactions and API calls
- Metadata: Stores and maintains persistent metadata for the system and data cubes
- Query Engine: Converts SQL queries into execution plans and communicates with storage layers
- Storage Engine: Accesses cube storage, typically using HBase by default
- Job Engine: Runs MapReduce or Spark tasks to build data cubes
Users
Prominent organizations like eBay, Meituan, Xiaomi, NetEase, Beike, and Yahoo! Japan employ Apache Kylin as their OLAP platform in production environments.
Roadmap
Significant achievements in Apache Kylin’s development roadmap include:
- Implementation of Hadoop 3.0 support featuring Erasure Coding - finalized (v2.5)
- Complete transition to a Spark-based Cube engine - finalized (v2.5)
- Broadened integration with data sources such as MySQL, Oracle, and SparkSQL - finalized (v2.6)
- Introduction of real-time analytics through Lambda Architecture - finalized (v3.0)
- Development of cloud-native storage utilizing Parquet format - underway (v4.0.0-alpha)
- Future plans for supporting ad hoc querying without the need for pre-built cubes