Apache Hadoop

Apache Hadoop
Company type	Open source
Industry	Software, Big data, Distributed computing
Founded	January 2006; 20 years ago
Founder	Doug Cutting, Mike Cafarella
Headquarters	United States
Products	Hadoop Common, HDFS, YARN, MapReduce, Ozone
Website	hadoop.apache.org

Apache Hadoop () represents a collection of open-source software designed for dependable, scalable, and distributed computing environments. It provides a software framework that facilitates distributed storage alongside the processing of big data through the MapReduce programming paradigm. Initially created to operate on computer clusters built with commodity hardware, Hadoop continues to be predominantly utilized in such settings, while also accommodating more advanced cluster configurations. Each Hadoop component is engineered with the expectation of frequent hardware malfunctions, incorporating built-in fault tolerance mechanisms.

Overview

At the heart of Apache Hadoop lie two primary elements: the Hadoop Distributed File System (HDFS), which handles data storage, and the MapReduce programming model, responsible for data processing. Hadoop divides files into sizable blocks that are spread across nodes within a cluster, sending packaged executable code to process these data segments concurrently. This approach takes advantage of data locality, resulting in faster and more efficient computation compared to traditional supercomputer systems that depend on parallel file systems and high-speed network connections.

The fundamental Apache Hadoop platform is composed of these key modules:

Hadoop Common – a set of libraries and utilities shared among other components;
Hadoop Distributed File System (HDFS) – a distributed file system storing information on commodity hardware, providing substantial aggregate bandwidth throughout clusters;
Hadoop YARN – launched in 2012, it orchestrates cluster resource management and schedules applications submitted by users;
Hadoop MapReduce – an implementation of the MapReduce framework for processing large-scale datasets;
Hadoop Ozone – introduced in 2020 as an object storage system tailored for Hadoop.

The term "Hadoop" frequently encompasses not only these core modules but also the extensive ecosystem of supplementary software installed alongside or atop the base framework. This ecosystem includes tools such as Apache Pig, Hive, HBase, Phoenix, Spark, ZooKeeper, Impala, Flume, Sqoop, Oozie, and Storm.

The foundational components of Apache Hadoop—MapReduce and HDFS—were inspired by Google's research papers on MapReduce and the Google File System. The majority of the framework is implemented in Java, supplemented by native C code and shell scripts. Additionally, Perl can be utilized through Hadoop Streaming to develop map and reduce functions.

History

The inception of Hadoop traces back to co-founders Doug Cutting and Mike Cafarella, who were influenced by the 2003 publication on the Google File System. They further expanded upon this foundation using Google's paper titled MapReduce: Simplified Data Processing on Large Clusters. Initially part of the Apache Nutch project, Hadoop was established as an independent subproject in January 2006. The name "Hadoop" was chosen by Doug Cutting while working at Yahoo!, inspired by his son's toy elephant. The original code comprised roughly 5,000 lines dedicated to HDFS and approximately 6,000 lines for MapReduce.

Owen O'Malley became the first contributor in March 2006, with the release of Hadoop 0.1.0 following shortly after in April 2006. Since then, development has progressed through contributions from a broad community. The initial design document for the Hadoop Distributed File System was authored by Dhruba Borthakur in 2007.

Architecture

Hadoop is composed of several components, including the Hadoop Common package that offers file system and operating system abstractions, a MapReduce processing engine (either MapReduce/MR1 or YARN/MR2), and the Hadoop Distributed File System (HDFS). The Common package contains the necessary Java Archive (JAR) files and scripts required to initiate Hadoop operations.

To optimize task scheduling, Hadoop-compatible file systems provide location awareness by identifying the rack or network switch associated with each worker node. This capability allows Hadoop applications to execute code on the node where data is stored or at least within the same rack, thereby reducing backbone network traffic. HDFS utilizes this feature for replicating data across different racks, enhancing fault tolerance and ensuring data remains accessible despite hardware failures such as power outages or network switch malfunctions.

A standard Hadoop cluster consists of one master node alongside multiple worker nodes. The master node hosts services including the Job Tracker, Task Tracker, NameNode, and DataNode. Worker nodes typically serve dual roles as DataNode and TaskTracker, although configurations with nodes dedicated solely to data storage or computation are possible for particular scenarios.

Hadoop requires Java Runtime Environment (JRE) version 1.6 or higher and relies on SSH configuration between nodes to facilitate standard startup and shutdown processes.

In more extensive installations, HDFS nodes are overseen by a dedicated NameNode server responsible for maintaining the file system metadata. This is supported by a Secondary NameNode that generates periodic snapshots to prevent corruption and data loss. A separate JobTracker manages job scheduling across the cluster. When Hadoop MapReduce operates with alternative file systems, the HDFS components are substituted by corresponding services specific to those file systems.

Hadoop Distributed File System

The Hadoop Distributed File System (HDFS) is a scalable, portable, distributed file system implemented in Java for use within the Hadoop ecosystem. Hadoop integrates HDFS for storing data and MapReduce for processing it.

HDFS comprises five primary services:

Name Node
Secondary Name Node
Job Tracker
Data Node
Task Tracker

The first three are classified as Master Services, while the latter two function as Slave Services. Communication occurs among Master nodes as well as among Slave nodes. The Name Node acts as the master managing metadata, whereas Data Nodes serve as slaves storing actual data blocks.

Name Node: This singular master node maintains metadata about files, including block locations and replication information, and directly interacts with client requests.

Data Node: Responsible for storing data blocks and handling client read/write operations. Each Data Node transmits a heartbeat signal every three seconds to the Name Node; if heartbeats cease, replication of affected data blocks is initiated elsewhere.

Secondary Name Node: Functions as a checkpointing node that assists the Name Node by creating snapshots of metadata files (fsimage and edit logs), facilitating faster recovery after failures.

Job Tracker: Receives MapReduce job submissions from clients and queries the Name Node to determine data locations.

Task Tracker: Operates as a slave node executing map or reduce tasks assigned by the Job Tracker on local data blocks.

Typically, a Hadoop cluster includes one Name Node accompanied by multiple Data Nodes. Redundancy mechanisms exist for safeguarding the critical Name Node. Data Nodes deliver data blocks over TCP/IP using an HDFS-specific protocol, while clients interact through remote procedure calls (RPC).

HDFS is designed to store very large files—ranging from gigabytes to terabytes—distributed across numerous machines. It ensures reliability via replication, usually set at a default factor of three copies distributed across different racks to improve fault tolerance. Data Nodes collaborate to balance storage and maintain replication levels. Although HDFS does not fully comply with POSIX standards, it emphasizes high throughput and supports operations such as appending data.

Since May 2012, HDFS has incorporated high-availability capabilities permitting manual failover of the Name Node to a standby server, with ongoing efforts aimed at enabling automatic failover functionality.

The Secondary Name Node is often misunderstood as a backup but actually serves to periodically capture metadata snapshots that accelerate recovery processes after failures. To address scalability challenges and mitigate single points of failure, HDFS Federation allows multiple namespaces managed by distinct Name Nodes; however, issues related to handling numerous small files, scalability limits, and metadata bottlenecks persist.

A significant benefit of HDFS lies in its awareness of data locality between Job Tracker and Task Tracker: tasks are preferentially scheduled on nodes where relevant data resides locally, reducing network load and improving job execution efficiency—an essential factor in processing large-scale datasets.

HDFS is optimized primarily for workloads involving mostly immutable files and lacks robust support for concurrent write operations. It can be mounted on Linux or Unix systems using Filesystem in Userspace (FUSE).

Access to HDFS can be achieved through native Java APIs, Thrift APIs supporting various programming languages, command-line tools, web user interfaces over HTTP, or third-party client applications.

Developed in Java for cross-platform compatibility across diverse hardware and operating systems, HDFS may experience performance trade-offs due to limited platform-specific optimizations inherent in its implementation.

Given its widespread enterprise use, monitoring HDFS performance at scale is crucial. Solutions such as Hortonworks Data Platform (HDP), Cloudera’s management tools, and Datadog provide comprehensive monitoring capabilities covering metrics from Data Nodes, Name Nodes, and underlying operating system components.

Alternative file systems

Hadoop is capable of working with any distributed file system that the operating system can mount using a file:// URL; however, this approach compromises data locality. To reduce network overhead, Hadoop depends on file system bridges that supply information about server proximity.

By May 2011, Apache Hadoop supported the following file systems:

HDFS: Hadoop’s proprietary rack-aware file system engineered to scale up to tens of petabytes, operating on top of underlying OS file systems.
Apache Hadoop Ozone: An object store compatible with HDFS, optimized for handling billions of small files.
FTP file system: Enables storage on remote FTP servers.
Amazon S3: Object storage service tailored for clusters running on Amazon Elastic Compute Cloud; it lacks rack-awareness because access occurs remotely.
Windows Azure Storage Blobs (WASB): Provides HDFS extension capabilities to access Azure blob storage without permanently relocating data into the cluster.

Numerous third-party file system bridges exist but are not bundled with the standard Hadoop releases. Nonetheless, certain commercial Hadoop distributions, particularly those from IBM and MapR, adopt alternative default file systems:

In 2009, IBM integrated Hadoop support over the IBM General Parallel File System and released the source code in October of that year.
Parascale made its Hadoop integration source code publicly available in April 2010.
Appistry introduced a Hadoop file system driver for its CloudIQ Storage solution in 2010.
HP presented a location-aware driver for the IBRIX Fusion file system during 2010.
In May 2011, MapR Technologies Inc. launched MapR FS, a substitute for HDFS that provides full random-access read/write functionality.

These initiatives demonstrate key vendor strategies aimed at boosting Hadoop’s storage adaptability and efficiency within enterprise settings.

JobTracker and TaskTracker: core MapReduce components

Sitting atop the file systems, the MapReduce Engine features a JobTracker responsible for receiving job submissions from clients. The JobTracker assigns tasks to available TaskTracker nodes with an emphasis on data locality. When using a rack-aware file system, the JobTracker prioritizes task allocation to the node containing the data; if unavailable, it selects nodes within the same rack to reduce network traffic. TaskTrackers that fail or time out cause tasks to be rescheduled. Each TaskTracker initiates separate Java virtual machine (JVM) instances per task to isolate failures effectively. TaskTrackers periodically send heartbeat signals to the JobTracker, whose status is accessible through a web interface powered by Jetty.

Notable limitations include:

Task assignment is based on fixed "slots" per TaskTracker without assessing current load conditions.
Slow-performing TaskTrackers can impede overall job completion, particularly toward the end; speculative execution addresses this by redundantly running tasks across multiple nodes.

Scheduling

By default, Hadoop employs FIFO scheduling with five priority tiers. Starting from version 0.19, the scheduler was decoupled from the JobTracker architecture, allowing alternative scheduling mechanisms such as the Fair scheduler and Capacity scheduler to be implemented.

Fair scheduler

Created by Facebook, the fair scheduler is designed to deliver rapid response times for smaller tasks while maintaining Quality of service (QoS) for critical production workloads. Its fundamental concepts include:

Organizing jobs into distinct pools.
Guaranteeing each pool a minimum allocation of resources.
Equitably distributing any surplus capacity among active jobs.

Jobs that are not categorized are placed into a default general pool. Each pool defines minimum allocations for map and reduce slots and restricts the number of jobs running simultaneously.

Capacity scheduler

Yahoo developed the capacity scheduler, which shares several characteristics with the fair scheduler:

Resource capacity is divided among queues as proportional shares.
Resources exceeding queue limits can be assigned to jobs.
Within a queue, jobs with elevated priority receive preferential access to resources.

Once a job begins execution, there is no preemption permitted.

Difference between Hadoop 1 and Hadoop 2 (YARN)

The primary advancement in Hadoop 2 is the introduction of YARN (Yet Another Resource Negotiator), which replaces the original MapReduce framework. YARN enhances resource management across applications by utilizing two key daemons: the resource manager responsible for tracking jobs and allocating resources, and the application master that oversees job execution.

Difference between Hadoop 2 and Hadoop 3

Hadoop 3 addresses the single point of failure problem found in Hadoop 2 by supporting multiple namenodes. It also integrates containerization technology akin to Docker, facilitating faster application deployment.

Notable improvements encompass decreased storage overhead via erasure coding and native GPU support within clusters, enabling efficient processing of deep learning tasks.

Other applications

Beyond MapReduce, HDFS serves as a foundation for projects such as the HBase database, Apache Mahout for machine learning, and Apache Hive for data warehousing. Hadoop is well-suited for batch-processing large-scale data workloads that benefit from parallelism. Additionally, it complements real-time processing frameworks like lambda architecture, Apache Storm, Flink, and Spark Streaming.

Commercial implementations of Hadoop include:

Analysis of logs and clickstream data
Marketing intelligence
Machine learning and data mining operations
Image processing workflows
Handling of XML messages
Web crawling activities
Archival compliance for relational and tabular datasets

Prominent use cases

On February 19, 2008, Yahoo! Inc. announced what was then considered the world’s largest Hadoop production environment. The Yahoo! Search Webmap operates on a Linux cluster with over 10,000 cores, processing data utilized in every Yahoo! web search query. Yahoo! maintains several Hadoop clusters, each independently bootstrapping its Linux image and Hadoop distribution. These clusters perform various tasks including index computations for Yahoo! search. In June 2009, Yahoo! contributed its Hadoop source code to the open-source community.

By 2010, Facebook reported owning the largest Hadoop cluster worldwide with 21 PB of storage capacity. This expanded to 100 PB by June 2012, with daily data growth nearing half a petabyte later that year.

By 2013, adoption of Hadoop had grown considerably, with over fifty percent of Fortune 50 companies incorporating it into their operations.

Hadoop hosting in the cloud

Hadoop can be implemented either on-premises or via cloud-based platforms. Utilizing cloud infrastructure allows organizations to leverage Hadoop capabilities without the need to purchase physical hardware or acquire specialized configuration skills.

Commercial support

A number of vendors offer commercial Hadoop solutions along with professional support services.

Branding

The Apache Software Foundation mandates that the designation Apache Hadoop or Distributions of Apache Hadoop applies exclusively to software officially released under the Apache Hadoop Project. The terminology surrounding compatibility and naming conventions remains a topic of debate within the developer community.

Papers

Influential publications shaping Hadoop and large-scale data processing include:

Jeffrey Dean and Sanjay Ghemawat (2004) MapReduce: Simplified Data Processing on Large Clusters, which served as the inspiration for Doug Cutting’s development of the open-source Hadoop platform.
Michael Franklin, Alon Halevy, and David Maier (2005) From Databases to Dataspaces: A New Abstraction for Information Management, highlighting adaptable data storage models and evolving API frameworks.
Fay Chang and colleagues (2006) Bigtable: A Distributed Storage System for Structured Data, from Google.
Robert Kallman et al. (2008) H-store: a high-performance, distributed main memory transaction processing system.

Bibliography

External links

Initial release date	Latest version	Release date
	0.10.1	2007-01-11
	0.11.2	2007-02-16
2007-03-02	0.12.3	2007-04-06
2007-06-04	0.13.1	2007-07-23
2007-09-04	0.14.4	2007-11-26
2007-10-29	0.15.3	2008-01-18
2008-02-07	0.16.4	2008-05-05
2008-05-20	0.17.2	2008-08-19
2008-08-22	0.18.3	2009-01-29
2008-11-21	0.19.2	2009-07-23
2009-04-22	0.20.205.0	2011-10-17
2011-05-11	0.21.0
2011-12-10	0.22.0
2011-11-11	0.23.11	2014-06-27
2011-12-27	1.0.4	2012-10-12
2012 -10 -13	1 .1 .2	2013 -02 -15
2013 -05 -13	1 .2 .1	2013 -08 -01
2012 -05 -23	2 .0 .6 -alpha	2013 -08 -23
2013 -08 -25	2 .1 .1 -beta	2013 -09 -23
2013 -12 -11	2 .2 .0
2014 -02 -20	2 .3 .0
2014 -04 -07	2 .4 .1	2014 -06 -30
2014 -08 -11	2 .5 .2	2014 -11 -19
2014 -11 -18	2 .6 .5	2016 -10 -08
2015 -04 -21	2 .7 .7	2018 -05 -31