Apache OODT
| Company type | Open source software |
|---|---|
| Industry | Data management software |
| Predecessor | NASA Jet Propulsion Laboratory project |
| Founder | Daniel J. Crichton, Sean Kelly, Steve Hughes |
| Fate | Contributed to the Apache Software Foundation |
Area served | NASA scientific archives, Big Data processing, bioinformatics |
Key people | Dr. Chris Mattmann, Dr. Justin Erenkrantz |
| Products | Data management system framework |
| Owner | Apache Software Foundation |
| Parent | Apache Software Foundation |
| Website | oodt |
The Apache Object Oriented Data Technology (OODT) is an open-source data management system framework overseen by the Apache Software Foundation. Initially created at the NASA Jet Propulsion Laboratory, OODT facilitates the capture, processing, and distribution of data within NASA's scientific archives.
History
OODT originated as a project within the NASA Jet Propulsion Laboratory, spearheaded by Daniel J. Crichton, Sean Kelly, and Steve Hughes, with an emphasis on integrating information and performing searches utilizing XML.
Following its implementation in the Planetary Data System and the National Cancer Institute's Early Detection Research Network (EDRN), OODT underwent significant development in 2005 to support extensive data processing needs for NASA's Orbiting Carbon Observatory (OCO) initiative. This expansion involved handling up to 10,000 daily jobs and managing several hundred terabytes of data, leading to a comprehensive redesign driven by Dr. Chris Mattmann from 2005 through 2009.
Leveraging his experience with projects such as Apache Nutch and Hadoop, Mattmann restructured OODT to conform with Apache Software Foundation guidelines. With assistance from Dr. Justin Erenkrantz, then ASF President, OODT was contributed to the Apache Foundation in 2009, marking it as the first NASA project under ASF governance. The software reached its 1.0 release seven years later.
Features
OODT is designed primarily for two key applications: large-scale Big Data processing and Information integration, as described in Mattmann’s publications from ICSE 2006 and SMC-IT 2009. It comprises three fundamental services:
File Manager
Oversees file metadata, tracks file locations, and facilitates transferring files from temporary staging areas to designated storage.
Workflow Manager
Controls both data and process flow for complex workflows, supporting reproducibility and enabling the construction of scientific pipelines.
Resource Manager
Distributes workflow tasks to suitable resources based on criteria such as Python availability or hardware specifications.
Building upon these services, OODT offers three client-facing frameworks:
File Crawler
Automatically extracts metadata by employing Apache Tika to detect file types and ingest relevant information into the File Manager.
Catalog and Archive Crawling Framework
Implements a push/pull mechanism to retrieve remote files and incorporate them into the system.
Catalog and Archive Service Production Generation Executive (CAS-PGE)
Encapsulates scientific algorithms for execution independent of environment, capturing provenance details and facilitating integration into production workflows.
Additional modules include:
CAS RESTful Services
APIs that provide access to functionalities of the File Manager, Workflow Manager, and Resource Manager.
OPSUI Monitor Dashboard
A web-based application developed with Apache Wicket that monitors OODT services through JAX-RS APIs.
The rationale behind OODT’s redesign was elaborated in a 2013 article in Nature (journal) authored by Mattmann titled A Vision for Data Science. Developed using Java, the platform also supports interoperability with other languages such as Python (programming language) via its REST API.
Noteworthy Applications
OODT has played a role in various NASA projects, including Soil Moisture Active Passive and New Horizons. Its capabilities also extend to supporting the Square Kilometre Array telescope, thereby broadening its use in fields such as Earth science, planetary research, and radio astronomy. Beyond space science, OODT is applied within bioinformatics and is integrated into the Knowledgent Big Data Platform.