Apache Hive
| Company type | Open-source software |
|---|---|
| Industry | Data warehousing, Big data analytics |
| Founder | |
Area served | Worldwide |
| Products | Data warehouse system, HiveQL query language |
| Owner | Apache Software Foundation |
| Parent | Apache Software Foundation |
| Website | hive |
Apache Hive is a data warehouse system developed atop Apache Hadoop to facilitate querying and analyzing data. It provides an SQL-like interface for querying information stored across multiple databases and file systems compatible with Hadoop. Traditionally, executing SQL queries on distributed data required programming through the MapReduce Java API.
Hive simplifies this process by enabling SQL-style queries (HiveQL) without the need for detailed Java coding, bridging SQL querying capabilities with the Hadoop ecosystem—commonly utilized in data warehousing scenarios. Originally created by Facebook, Apache Hive is currently supported and improved by organizations such as Netflix and the Financial Industry Regulatory Authority (FINRA). Additionally, Amazon offers its own Hive variant within the Amazon Elastic MapReduce platform on Amazon Web Services.
Features
Apache Hive supports large-scale data analysis of datasets stored in Hadoop’s HDFS as well as compatible storage systems like Amazon S3 and Alluxio. Its query language, HiveQL, resembles SQL and operates under a schema-on-read model, translating queries into jobs executed by MapReduce, Apache Tez, or Spark. These processing engines run under Hadoop’s resource manager, YARN. Although index support was once available to accelerate query execution, this capability was discontinued starting from version 3.0.
Notable capabilities include:
- Compatibility with multiple storage formats including plain text, RCFile, HBase, ORC, among others.
- Metadata maintained within a relational database management system, enhancing semantic validation during query processing.
- Handling of compressed data using algorithms such as DEFLATE, BWT, and Snappy.
- Availability of built-in and customizable user-defined functions (UDFs) to extend data processing functionality.
- Automatic conversion of HiveQL statements into MapReduce, Tez, or Spark execution plans.
By default, Hive utilizes an embedded Apache Derby database for metadata storage but can be configured to use client/server databases like MySQL.
Initially, supported file formats included plain text, sequence file, optimized row columnar (ORC), and RCFile. Support for Apache Parquet was introduced via plugin after version 0.10 and became integrated natively from version 0.13 onward.
Architecture
The core components that constitute Hive’s architecture are:
- Metastore: Responsible for storing metadata about tables—including schema details and data locations—as well as partition metadata that tracks how data is distributed across cluster nodes. This metadata resides in a conventional RDBMS and is regularly backed up to avoid loss.
- Driver: Functions as the main controller that accepts HiveQL commands, initiates query execution by establishing sessions, monitors progress, and records execution metadata. It also collects the final query results following the Reduce phase.
- Compiler: Translates HiveQL queries into an executable plan by first building an abstract syntax tree (AST), performing error checks, then converting it into a directed acyclic graph (DAG) which associates operators with specific MapReduce stages and tasks.
- Optimizer: Enhances the execution plan’s efficiency by applying transformations such as join merging or task decomposition for scalability. This component’s logic can be modified or replaced; Apache Hive includes YSmart, an optimizer that consolidates related MapReduce jobs into a single job to significantly reduce runtime.
- Executor: Manages task execution by communicating with Hadoop’s job tracker to schedule tasks and efficiently pipeline dependent operations.
- CLI, UI, and Thrift Server: Provides various user interfaces for submitting queries and monitoring their status. The Thrift server allows external clients to connect remotely using protocols like JDBC or ODBC.
HiveQL
Although HiveQL is derived from SQL, it does not entirely conform to the SQL-92 specification. It incorporates enhancements such as multi-table inserts and the ability to create tables as select. In its early versions, HiveQL did not support transactions or materialized views and had restricted subquery functionality. Comprehensive ACID compliance, including insert, update, and delete operations, was introduced starting with version 0.14.
HiveQL queries are translated internally into a directed acyclic graph composed of MapReduce, Tez, or Spark tasks that run on the Hadoop platform.
Example
The following word count example illustrates how HiveQL processes data:
<syntaxhighlight lang="googlesql" line> DROP TABLE IF EXISTS docs; CREATE TABLE docs (line STRING); LOAD DATA INPATH 'input_file' OVERWRITE INTO TABLE docs; CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM
(SELECT explode(split(line, '\s')) AS word FROM docs) temp
GROUP BY word ORDER BY word; </syntaxhighlight>
Explanation of this example: <syntaxhighlight lang="googlesql" line> DROP TABLE IF EXISTS docs; CREATE TABLE docs (line STRING); </syntaxhighlight> Removes the table if present and defines a new table with a single column.
<syntaxhighlight lang="googlesql" line start="3"> LOAD DATA INPATH 'input_file' OVERWRITE INTO TABLE docs; </syntaxhighlight> Imports data from the specified input_file, replacing any existing contents in the table.
<syntaxhighlight lang="googlesql" line start="4" highlight="6"> CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) temp GROUP BY word ORDER BY word; </syntaxhighlight> Generates a new table with columns for words and their respective counts by splitting lines into individual words, aggregating counts per word, and ordering results alphabetically.
Comparison with traditional databases
While Hive’s data storage and query mechanisms share similarities with conventional databases, its operation is distinct due to its reliance on the Hadoop framework and limitations inherent in MapReduce.
Conventional databases implement schema on write, enforcing data structure validation during data ingestion to ensure quality and optimize query speed. Conversely, Hive employs schema on read, where data is checked only when queried. This approach allows rapid initial data loading and greater adaptability when schemas are defined dynamically but may lead to slower query execution.
Transactions are fundamental in traditional RDBMS environments. Hive offers full support for ACID properties—Atomicity, Consistency, Isolation, and Durability. Initially, Hive 0.13 supported transactions limited to partitions; however, version 0.14 introduced comprehensive row-level transaction capabilities for INSERT, UPDATE, and DELETE. Activating these features requires setting configuration parameters such as hive.support.concurrency, hive.enforce.bucketing, and hive.exec.dynamic.partition.mode.
Security
With the release of Hive version 0.7.0, Hadoop security was enhanced by incorporating Kerberos authentication, facilitating verified identity exchange between clients and servers. This improvement resolved earlier security issues where users could impersonate others through the hadoop.job.ugi configuration, and MapReduce jobs were previously executed under a shared user account. Following this upgrade, TaskTracker processes run using the credentials of the initiating user, effectively preventing identity spoofing. Access controls within Hive rely on the HDFS authorization framework, which enforces permissions for users, groups, and others across read, write, and execute actions. The default file permission mask for newly created files is adjustable through the Hive parameter hive.files.umask.value.