Apache ZooKeeper
| Company type | Open-source software project |
|---|---|
| Industry | Distributed computing, Cloud computing |
| Founder | Apache Software Foundation |
Area served | Worldwide |
| Products | Distributed coordination service, hierarchical key-value store |
| Owner | Apache Software Foundation |
| Website | zookeeper |
Apache ZooKeeper is an open-source server designed to provide dependable distributed coordination for cloud-based applications, developed under the auspices of the Apache Software Foundation.
ZooKeeper offers a hierarchical key-value store that supports distributed configuration management, synchronization, and naming functionalities within extensive systems. Initially part of the Hadoop project, it has since evolved into a standalone top-level Apache initiative.
Overview
The design of ZooKeeper ensures high availability by employing redundant services, enabling clients to connect to alternative leaders if the primary one becomes unavailable. Data is organized in a hierarchical namespace resembling a file system, which facilitates shared configuration management. The platform utilizes the ZooKeeper Atomic Broadcast (ZAB) protocol to maintain strict ordering of updates.
Prominent organizations leveraging ZooKeeper include Yelp, Rackspace, Yahoo!, Reddit, NetApp SolidFire, Meta, Twitter, and eBay. Additionally, it supports open-source enterprise search solutions such as Solr and distributed databases like Apache Pinot.
Inspired by Google's Chubby lock service, ZooKeeper was originally created at Yahoo! to oversee process management on large-scale data clusters by storing status information on ZooKeeper servers that interact with clients. It was developed to overcome challenges faced when deploying distributed big-data applications.
Primary attributes of Apache ZooKeeper:
- Dependable: maintains operation despite node failures.
- Straightforward Architecture: a shared hierarchical namespace that simplifies process coordination.
- High Performance: optimized for environments with predominantly read operations.
- Scalable: enhanced performance achieved by adding more nodes.
Architecture
The fundamental elements within ZooKeeper’s architecture are:
- Node: individual machines within the cluster.
- ZNode: designated points where cluster state updates are recorded.
- Client applications: software that interfaces with distributed systems.
- Server applications: components offering a unified interface for clients.
Services operate on an ensemble of servers, each holding an in-memory replica of the complete data tree alongside persistent transaction logs and snapshots. Clients establish TCP connections to submit requests, send heartbeats, and receive responses as well as watch events for monitoring purposes.
Use cases
Common applications of ZooKeeper include:
- Naming service
- Configuration management
- Data synchronization
- Leader election
- Message queue
- Notification system
Client libraries
In addition to its native client libraries, ZooKeeper benefits from third-party tools such as Apache Curator and Kazoo, which improve usability, introduce additional features, and expand support across various programming languages.
Apache projects using ZooKeeper
ZooKeeper serves as a foundational component for numerous significant Apache projects:
- Apache Hadoop
- Apache Accumulo
- Apache HBase
- Apache Hive
- Apache Kafka (up to version 4.0.0)
- Apache Drill
- Apache Solr
- Apache Spark
- Apache NiFi
- Apache Druid
- Apache Helix
- Apache Pinot
- Apache Bookkeeper
- Apache Pulsar