Quantcast File System
| Company type | Open-source software |
|---|---|
| Industry | Distributed file system |
| Predecessor | Kosmos File System (KFS) |
| Founded | 2012 |
| Founder | Quantcast |
| Products | Distributed file system software |
Quantcast File System (QFS) is an open-source distributed file system engineered to support extensive MapReduce and batch-processing tasks. It serves as a high-performance and cost-effective substitute for the Apache Hadoop Distributed File System (HDFS), specifically designed for large-scale processing clusters.
Design
QFS operates on clusters composed of hundreds to thousands of standard Linux servers, which it consolidates into a single storage entity. The system consists of three primary elements:
- Each data-hosting node runs a chunk server responsible for managing input/output operations and tracking storage usage.
- A centralized metaserver oversees the directory hierarchy and maps files to storage locations, orchestrating chunk servers while continuously monitoring system status. To optimize speed, metadata is kept in memory, supplemented by disk-based checkpoints and logs to facilitate recovery.
- The client module offers a file system application programming interface (API), obtaining data location details from the metaserver and performing direct read/write interactions with chunk servers.
Reliability is ensured through Reed–Solomon error correction, which duplicates data across nine different machines—six containing data stripes and three holding parity information. This setup allows data recovery even if up to three machines fail, maintaining fault tolerance with a 50% storage overhead.
Implemented in C++, QFS maintains a consistent memory footprint and uses direct input/output (I/O) techniques to deliver efficient operation.
History
The origins of QFS trace back to the Kosmos File System (KFS), an open-source initiative started by Kosmix in 2005. In 2007, Quantcast took over KFS and progressively improved it, culminating in the public release of QFS 1.0 as an open-source project in September 2012.
References