Roadrunner (supercomputer)
| Company type | Supercomputer project |
|---|---|
| Industry | High-performance computing |
| Founded | 2002 |
| Defunct | March 31, 2013 |
| Fate | Retired and decommissioned |
| Successor | Cielo system |
| Headquarters | Los Alamos , United States |
| Owner | IBM, U.S. Department of Energy (NNSA) |
| Parent | IBM |
Roadrunner was a supercomputer engineered by IBM specifically for the Los Alamos National Laboratory located in New Mexico, USA. With an investment of US$100 million, the machine was engineered to reach a peak capacity of 1.7 petaflops. On May 25, 2008, it attained 1.026 petaflops, marking it as the first system worldwide on the TOP500 list to sustain over 1.0 petaflops on the LINPACK test.
By November 2008, Roadrunner's peak performance increased to 1.456 petaFLOPS, securing its leading rank on the TOP500 rankings. It also earned recognition as the fourth most energy-efficient supercomputer globally according to the Supermicro Green500 list, delivering an efficiency of 444.94 megaflops per watt. The machine’s pioneering hybrid architecture significantly influenced later developments in energy-conscious supercomputing. Roadrunner was retired on March 31, 2013, and succeeded by the Cielo system, which was installed in 2010.
Overview
The U.S. Department of Energy's (DOE) National Nuclear Security Administration (NNSA) commissioned IBM to create the Roadrunner supercomputer. Its architecture was hybrid, integrating 12,960 IBM PowerXCell 8i processors alongside 6,480 AMD Opteron dual-core CPUs within custom-built blade servers networked through InfiniBand. Roadrunner operated using both Red Hat Enterprise Linux and Fedora, with management handled by the distributed computing tool xCAT. The system employed the Open MPI implementation of the Message Passing Interface standard for parallel processing.
Spanning approximately 296 server racks over , Roadrunner became fully functional in 2008 and remained active until its decommissioning in March 2013. The DOE primarily utilized it to simulate aging processes in nuclear materials to verify the safety and reliability of the United States’ nuclear arsenal. Beyond this mission, Roadrunner supported computational tasks in scientific research, financial modeling, automotive engineering, and aerospace development.
Hybrid design
Roadrunner distinguished itself from other supercomputers of its era by adopting a hybrid design philosophy that traces back to Seymour Cray’s innovations with the CDC 6600 in 1964 and CDC 7600 in 1969. Unlike earlier hybrids where peripheral processors were limited to operating system functions while applications ran solely on a central CPU, Roadrunner required all applications to be explicitly developed for its hybrid setup. Its architecture combined dual-core AMD Opteron processors based on AMD64 architecture with IBM’s proprietary PowerXCell 8i processors attached directly to each Opteron core. Essentially, Roadrunner operated as a cluster of Opteron CPUs enhanced by Cell processors, with each node pairing one Cell processor alongside an Opteron core.
Development
Roadrunner’s development commenced in 2002, with initial activation occurring in 2006 and full operational status achieved by 2008 after three distinct construction phases reflecting its innovative and complex design. Its forerunner was Dark Horse, an early hybrid prototype developed at Los Alamos that initially incorporated ARM processors before transitioning to Cell processors while integrating advancements in 3D memory, networking, and processing technologies.
Phase 1
The first phase entailed constructing a conventional Opteron cluster to evaluate the viability of a hybrid configuration. This foundational system reached a performance level of 71 teraflops and became fully operational at Los Alamos National Laboratory in 2006.
Phase 2
Referred to as AAIS (Advanced Architecture Initial System), Phase 2 entailed constructing a compact hybrid prototype utilizing an earlier iteration of the Cell processor. This stage concentrated on creating software tailored for the hybrid design and became operational in January 2007.
Phase 3
The objective of Phase 3 was to surpass a sustained performance threshold of 1 petaflop by incorporating additional Opteron nodes alongside the latest PowerXCell processors, which delivered quintuple the power compared to those in Phase 2. Built at IBM’s facility in Poughkeepsie, New York, this system surpassed the 1 petaflop milestone on May 25, 2008. By summer 2008, the entire setup was transferred to its designated location in New Mexico.
Processors
Roadrunner integrated two distinct processor types: the AMD Opteron 2210 running at 1.8 GHz, deployed across computational and operational nodes totaling 6,912 units (6,480 dedicated to computation and 432 for operations), interconnected through HyperTransport, delivering a combined total of 13,824 cores; and the IBM PowerXCell 8i operating at 3.2 GHz, each containing one primary processing element (PPE) and eight specialized processing elements (SPEs). The system housed 12,960 PowerXCell processors, amounting to 116,640 cores.
TriBlade
The TriBlade unit logically merges two dual-core Opteron processors with 16 GB of RAM and four PowerXCell 8i CPUs alongside 16 GB of Cell RAM.
In terms of hardware, it consists of one LS21 Opteron blade, an expansion blade, and a pair of QS22 Cell blades. The LS21 blade contains two dual-core Opterons clocked at 1.8 GHz with a total of 16 GB memory (split as 8 GB per CPU). Each QS22 blade is equipped with two PowerXCell 8i CPUs running at 3.2 GHz and holds 8 GB memory (4 GB per CPU). The expansion blade connects the QS22 blades to the LS21 via four PCIe x8 links and facilitates external access through an InfiniBand 4x DDR adapter. Occupying four slots in total, three TriBlades can be installed within a single BladeCenter H chassis. Connectivity between the expansion blade and the Opteron is achieved using HyperTransport.
Connected Unit (CU)
A Connected Unit is composed of 60 BladeCenter H chassis containing a total of 180 TriBlades, all interconnected through a 288-port Voltaire ISR2012 Infiniband switch. Each CU accesses the Panasas file system via twelve System x3755 servers.
Specifications for each CU include:
- A total of 360 dual-core Opteron processors with an aggregate of 2.88 TiB RAM.
- 720 PowerXCell 8i cores supported by 2.88 TiB RAM.
- Twelve System x3755 servers equipped with dual 10-GBit Ethernet interfaces.
- A single Voltaire ISR2012 switch with 288 ports featuring 192 Infiniband 4x DDR connections servicing both the 180 TriBlades and twelve I/O nodes.
Roadrunner cluster
The entire Roadrunner cluster consists of eighteen Connected Units linked together by eight second-tier Infiniband ISR2012 switches. Each CU connects through twelve uplinks per switch, resulting in a total of ninety-six uplinks.
Cluster details:
- Totaling 6,480 Opteron processors paired with 51.8 TiB RAM distributed over 3,240 LS21 blades.
- Incorporating 12,960 Cell processors supported by another 51.8 TiB RAM across 6,480 QS22 blades.
- Hosting 216 System x3755 I/O nodes.
- Utilizing twenty-six ISR2012 Infiniband switches (each with a capacity of 288 ports).
- Spanning across two hundred ninety-six racks.
- Consuming approximately 2.345 MW of power.
Shutdown
IBM Roadrunner was officially retired on March 31, 2013. Although it was counted among the world's top-performing supercomputers, its power efficiency was relatively low—achieving 444 megaflops per watt in contrast to another system's 886 megaflops per watt. Before being taken offline, scientists carried out tests on memory and data flow to guide the development of future high-performance computing systems.
Following its decommissioning, Los Alamos National Laboratory shredded Roadrunner’s electronic parts due to the sensitive nature of its operations. However, certain elements were retained for archival and historical purposes.