Cascading (software)
| Company type | Open source software |
|---|---|
| Industry | Software, Big Data |
| Predecessor | Concurrent, Inc (now Driven, Inc) |
| Founder | Chris Wensel |
| Products | Software abstraction layer for Apache Hadoop and Apache Flink |
| Parent | Driven, Inc |
| Website | www |
Cascading serves as a software abstraction layer designed for Apache Hadoop and Apache Flink, facilitating the development and execution of intricate data workflows on Hadoop clusters through any JVM-compatible language such as Java, JRuby, or Clojure. It reduces the complexity involved in crafting MapReduce jobs. Cascading is distributed as open source under the Apache License, with commercial support provided by Driven, Inc.
Initially created by Chris Wensel, the founder of Concurrent, Inc (now Driven), Cascading continues to be actively supported by its community and offers multiple add-on modules.
Architecture
Cascading operates on top of Apache Hadoop, embedding its libraries within the Hadoop job .jar file. The framework comprises a data processing API, an integration API, a process planner, and a scheduler component.
By abstracting common data operations away from the underlying map and reduce tasks, Cascading takes advantage of Hadoop’s scalability. Developers define their workflows in a .jar following a ‘source-pipe-sink’ architecture: data originates from sources, passes through reusable ‘pipes’ for processing, and is ultimately directed to output ‘sinks’. These pipes are data-agnostic and can be combined into ‘flows’, which themselves can be organized into ‘cascades’. The scheduler manages these cascades to maintain dependency order. This modular approach supports reuse and flexible rearrangement to accommodate varied business needs.
Programming is done using JVM languages without requiring expertise in MapReduce, enabling seamless regression testing and integration with external systems similar to standard Java applications.
Cascading finds extensive use in areas such as targeted advertising, log analytics, bioinformatics research, machine learning workflows, predictive analytics, web content mining, and extract-transform-load (ETL) operations.
Uses of Cascading
In 2011, SD Times ranked Cascading among the top five most influential Hadoop projects. It has become a key open source tool within bioinformatics. The framework is highlighted in Hadoop: The Definitive Guide by Tom White and frequently referenced in presentations, academic conferences, and Hadoop user group forums as an effective solution for working with Hadoop and Apache Spark.
Prominent commercial implementations built on Cascading include:
- MultiTool on Amazon Web Services
- LogAnalyzer supporting Amazon CloudFront
- BackType – platform for social analytics
- Etsy – e-commerce marketplace
- FlightCaster – service predicting flight delays
- Ion Flux – DNA sequencing data analysis
- RapLeaf – systems for personalization and recommendations
- Razorfish – digital marketing agency
These deployments demonstrate Cascading’s significant role in big data processing across sectors like e-commerce, advertising, bioinformatics, and cloud computing services.
Domain-Specific Languages Built on Cascading
- PyCascading – developed by Twitter; repository hosted on GitHub
- Cascading.jruby – created by Gregoire Marabout; available on GitHub
- Cascalog – authored by Nathan Marz; accessible on GitHub
- Scalding – Scala interface for Cascading developed by Twitter; aids transition to Spark; hosted on GitHub