High Performance Storage System
| Company type | Collaboration |
|---|---|
| Industry | Data storage software |
| Founded | 1992 |
| Headquarters | United States |
Area served | Worldwide |
Key people | IBM Global Business Services, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, Lawrence Berkeley National Energy Research Scientific Computing Center, Oak Ridge National Laboratory, Sandia National Laboratory |
| Products | High Performance Storage System (HPSS) |
High Performance Storage System (HPSS) is a software-defined, policy-driven hierarchical storage management (HSM) solution designed for scalability, developed through the efforts of the HPSS Collaboration. It offers scalable archive, HSM, and file system functionalities by integrating cluster computing, LAN, and storage area network (SAN) technologies to harness the combined capacity and speed of multiple servers, disk arrays, tape drives, and automated tape libraries.
Architecture
HPSS provides support for a range of data access protocols such as FTP, parallel FTP, FUSE on Linux platforms, along with a comprehensive client API that enables parallel input/output operations.
Starting with version 7.5, HPSS delivers full compatibility with Linux, while the client API remains interoperable across AIX, Linux, and Solaris. The architecture is centered on IBM’s scalable relational database management system, Db2.
The HPSS Collaboration
The origins of HPSS trace back to 1992 when multiple United States Department of Energy (DOE) National Laboratories—Lawrence Livermore (LLNL), Los Alamos (LANL), Oak Ridge (ORNL), and Sandia (SNL)—joined forces with IBM to establish the National Storage Laboratory (NSL). The NSL’s mission was to commercialize solutions addressing critical limitations in computing and data storage performance. This initiative evolved into the HPSS collaboration involving IBM’s Houston Global Services and five DOE laboratories (Lawrence Berkeley [LBL], LLNL, LANL, ORNL, SNL).
Anticipating a surge in data volumes driven by computational capabilities reaching teraflop and petaflop scales, the collaboration engineered HPSS to handle petabyte-scale storage with transfer speeds in the gigabytes-per-second range and daily data throughput measured in tens of terabytes. The design target was a system capable of scaling by factors exceeding 1,000 to accommodate future growth.
The collaboration is founded on the understanding that no single organization can independently bridge the widening gap between advancing computational power and storage capacity. Over twenty global entities—including industry leaders, DOE laboratories, federal agencies, academic institutions, NSF supercomputing centers, the French Commissariat à l'Énergie Atomique et aux Énergies Alternatives (CEA), and Gleicher Enterprises—have contributed to HPSS development.
As of 2022, the core development consortium comprises:
- IBM Global Business Services (Houston, TX)
- Los Alamos National Laboratory (Los Alamos, NM)
- Lawrence Livermore National Laboratory (Livermore, CA)
- Lawrence Berkeley National Energy Research Scientific Computing Center (Berkeley, CA)
- Oak Ridge National Laboratory (Oak Ridge, TN)
- Sandia National Laboratory (Albuquerque, NM)
Notable achievements
- In March 2024, the ECMWF became the inaugural HPSS installation to maintain an exabyte of data within a single namespace.
- On November 14, 2007, a collaborative demonstration by the San Diego Supercomputer Center, IBM, DataDirect, and Brocade successfully executed a "Billion File" backup test from GPFS into HPSS.
- In May 2013, a 380 Petabyte-scale HPSS deployment was established at the National Center for Supercomputing Applications (NCSA) located at the University of Illinois at Urbana-Champaign.