Apache Pig
| Company type | Open source software project |
|---|---|
| Industry | Big data, Data processing |
| Founded | 2006 |
| Founder | Yahoo Research |
| Headquarters | United States |
| Products | Pig Latin (programming language) |
| Parent | Apache Software Foundation |
| Website | pig |
Apache Pig serves as a high-level framework designed for developing programs that operate on Apache Hadoop. Its scripting language, Pig Latin, facilitates the execution of Hadoop tasks through MapReduce, Apache Tez, or Apache Spark. By abstracting the complexity of Java MapReduce coding, Pig Latin offers a high-level syntax reminiscent of SQL used in relational database management. It also accommodates user-defined functions (UDFs) written in various programming languages such as Java, Python, JavaScript, Ruby, and Groovy.
History
Originating from Yahoo Research around 2006, Apache Pig was created to provide researchers with an efficient, flexible approach for developing and running MapReduce jobs on extensive datasets. The project was donated to the Apache Software Foundation in 2007.
| Version | Initial release date | Most recent version | Release date | |
|---|---|---|---|---|
| Template:Version | 2008-09-11 | 0.1.1 | 2008-12-05 | |
| Template:Version | 2009-04-08 | 0.2.0 | 2009-04-08 | |
| Template:Version | 2009-06-25 | 0.3.0 | 2009-06-25 | |
| Template:Version | 2009-08-29 | 0.4.0 | 2009-08-29 | |
| Template:Version | 2009-09-29 | 0.5.0 | 2009-09-29 | |
| Template:Version | 2010-03-01 | 0.6.0 | 2010-03-01 | |
| Template:Version | 2010-05-13 | 0.7.0 | 2010-05-13 | |
| Template:Version | 2010-12-17 | 0.8.1 | 2011-04-24 | |
| Template:Version | 2011-07-29 | 0.9.2 | 2012-01-22 | |
| Template:Version | 2012-01-22 | 0.10.1 | 2012-04-25 | |
| Template:Version | 2013-02-21 | 0.11.1 | 2013-04-01 | |
| Template:Version | 2013-10-14 | 0.12.1 | 2014-04-14 | |
| Template:Version | 2014-07-04 | 0.13.0 | 2014-07-04 | |
| Template:Version | 2014-11-20 | 0.14.0 | 2014-11-20 | |
| Template:Version | 2015-06-06 | 0.15.0 | 2015-06-06 | |
| Template:Version | 2016-06-08 | 0.16.0 | 2016-06-08 | |
| Template:Version | 2017-06-19 | 0.17.0 | 2017-06-19 | |
| Template:Smalldiv | ||||
Naming
The designation "Pig" for the programming language was selected arbitrarily to ensure it was distinctive, memorable, and easy to spell. Initially referred to simply as "the language," the developers spontaneously adopted "Pig," which inspired related names such as Pig Latin (the language itself), Grunt (the command shell), and PiggyBank (a shared code repository). This unconventional branding has helped establish a unique presence within the big data ecosystem.
Example
An example of a typical Pig Latin "word count" script:
<syntaxhighlight lang=pig>
input_lines = LOAD '/tmp/my-copy-of-all-pages-on-internet' AS (line:chararray); words = FOREACH input_lines GENERATE FLATTEN(TOKENIZE(line)) AS word; filtered_words = FILTER words BY word MATCHES '\\w+'; word_groups = GROUP filtered_words BY word; word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS count, group AS word; ordered_word_count = ORDER word_count BY count DESC; STORE ordered_word_count INTO '/tmp/number-of-words-on-internet';
</syntaxhighlight>
This code initiates parallelized operations across a Hadoop cluster to tally words within vast datasets, illustrating Pig's capability for scalable big data processing and analysis.
Pig versus SQL
Pig utilizes a nested relational data model, supports lazy evaluation, and offers ETL functionalities along with the ability to store data at various points within the processing pipeline. It also enables the definition of execution plans and manages workflow branching through directed acyclic graphs (DAGs), instead of relying on straightforward linear pipelines. Although conventional DBMS systems can perform queries more rapidly after data is loaded—benefiting from features such as columnar storage, data compression, indexing, and transactional fault tolerance—the initial data loading phase may be slower compared to Pig's MapReduce-based processing.
Pig Latin operates in a procedural manner that complements pipeline-style processing, granting users explicit control over execution strategies and join operations. Conversely, SQL is a declarative language that specifies the desired output without detailing how joins or workflow splits should be carried out. Pig scripts define DAGs that support sophisticated workflows and allow embedding custom code at any stage, providing flexibility in pipeline construction without the necessity of first importing data into a database system.
These characteristics make Pig especially advantageous for extensive data transformation and processing activities within distributed computing environments where precise management of execution flow and seamless integration with user-defined code are essential.
See also
- Apache Hive
- Sawzall — Google's comparable tool