Titan (supercomputer)
| Company type | Supercomputer |
|---|---|
| Industry | High-performance computing |
| Predecessor | Jaguar |
| Founded | 2011 |
| Founder | Cray |
| Defunct | 2019 |
| Fate | Replaced by Summit |
| Headquarters | Oak Ridge, Tennessee , United States |
Area served | Scientific research community |
| Products | Supercomputing services |
| Owner | Oak Ridge National Laboratory (Department of Energy) |
Titan, also identified as OLCF-3, was a supercomputer engineered by Cray for the Oak Ridge National Laboratory to support a wide array of scientific investigations. As an enhancement of the earlier Jaguar system, Titan incorporated graphics processing units (GPUs) alongside conventional central processing units (CPUs), marking it as the first hybrid supercomputer to surpass 10 petaFLOPS. The upgrade process commenced in October 2011, followed by stability assessments in late 2012, and it became accessible to researchers by early 2013. The $60 million modernization effort was chiefly financed by the United States Department of Energy.
In 2019, Titan was succeeded at Oak Ridge by Summit, developed by IBM. Summit featured fewer nodes but delivered greater GPU power per node and incorporated advanced local non-volatile caching to enhance file system throughput.
Titan’s architecture combined 18,688 AMD Opteron CPUs with an equal count of Nvidia Tesla GPUs, delivering a theoretical peak performance of 27 petaFLOPS and achieving 17.59 petaFLOPS on the LINPACK benchmark. This capability earned it the top position on the November 2012 TOP500 rankings until it was overtaken by Tianhe-2 in June 2013.
Access to Titan was granted based on the importance of research projects and their compatibility with its hybrid design. Applications were required to be runnable on other supercomputers to prevent exclusive dependence on Titan. Early pilot projects concentrated on molecular-scale physics and climate simulations, with an additional 25 projects lined up. The integration of GPUs compelled developers to enhance code parallelism, frequently leading to performance improvements even on CPU-only platforms.
History
The initiative for a 20 petaFLOPS supercomputer at the Oak Ridge Leadership Computing Facility (OLCF) began in 2005 with Jaguar. Plans called for Titan’s replacement by a roughly 200 petaFLOPS system around 2016, supporting ORNL’s ambition for an exascale machine by 2020. Rather than constructing a new 15,000 m² facility, Titan utilized Jaguar’s existing infrastructure. The final system design was confirmed in 2010 following a 2009 agreement with Nvidia for GPU integration. Titan was initially revealed privately at SC10 in November 2010 and publicly announced in October 2011 as its upgrade phase commenced.
Jaguar itself evolved from the Cray XT3 platform (25 teraFLOPS) through XT4 (263 teraFLOPS) by 2008, and then XT5 (1.4 petaFLOPS) in 2009, ultimately achieving 1.76 petaFLOPS after its last enhancements.
Primarily funded by the US Department of Energy via ORNL, Titan’s GPU nodes received partial financial support from the National Oceanic and Atmospheric Administration (NOAA) in exchange for computing time allocation. The total initial expenditure was approximately $60 million, with NOAA contributing less than $10 million; the complete Cray contract amounted to $97 million excluding subsequent upgrades.
The year-long upgrade began in October 2011, transforming Jaguar’s cabinets into XK7 blades while maintaining operational status. Network bandwidth increased from 10 Gbit/s to 100 Gbit/s, and the interconnect technology shifted from Seastar to Gemini, enabling a direct three-dimensional torus interconnect network with wormhole flow control. System memory capacity doubled to 584 TiB. Initially, 960 nodes were outfitted with Fermi GPUs since Kepler GPUs were not yet available; these TitanDev nodes facilitated code testing and raised peak performance to 3.3 petaFLOPS. By September 2012, all Fermi GPUs were replaced with Nvidia K20X GPUs, completing the upgrade and prompting the system’s renaming to Titan in October.
In March 2013, Nvidia introduced the consumer-grade GTX Titan card employing the same GPU die as Titan’s K20X GPUs. Early acceptance testing of Titan revealed motherboard solder cracking due to excessive gold in PCIe connectors; Cray assumed repair costs and restored functionality incrementally across cabinets. By April 2013 repairs were complete and full acceptance was achieved by June.
Titan’s theoretical maximum hardware performance reached up to 27 petaFLOPS under ideal software conditions. It topped the November 2012 TOP500 list at 17.59 petaFLOPS, surpassing IBM Sequoia, and secured third place on the Green500 list for energy efficiency. By June 2013, Titan had fallen to second place behind Tianhe-2 on TOP500 and dropped to twenty-ninth on Green500 but retained its theoretical peak of 27 petaFLOPS without undergoing retesting.
Hardware
Titan operates within Jaguar's 200 cabinets, occupying a total area of 404 square meters (4,352 ft2), but features enhanced internal components and upgraded networking infrastructure. By retaining Jaguar’s existing power and cooling infrastructure, approximately $20 million was saved. The system receives power through three-phase 480 V lines, which allow for the use of thinner cables compared to the US standard 208 V, resulting in a $1 million reduction in copper costs. At maximum load, Titan consumes 8.2 MW—1.2 MW more than Jaguar—while delivering almost tenfold the floating-point computational capacity. In the event of a power outage, carbon fiber flywheel energy storage sustains networking and storage operations for up to 16 seconds; diesel generators then activate within 2 seconds and achieve full output in 7 seconds, providing continuous backup for networking and storage systems, though not for processing tasks.
The system comprises 18,688 nodes, configured as four nodes per blade and 24 blades per cabinet. Each node contains a 16-core AMD Opteron 6274 processor, 32 GB of DDR3 ECC memory, and an Nvidia Tesla K20X GPU equipped with 6 GB of GDDR5 ECC memory. Collectively, Titan houses 299,008 processor cores and a combined CPU/GPU memory capacity of 693.6 TiB.
Initially, Titan utilized Jaguar’s Lustre storage system with a capacity of 10 PB and a bandwidth of 240 GB/s. In April 2013, this was expanded to 40 PB with an enhanced transfer speed of 1.4 TB/s. GPUs were chosen for their superior parallel processing capabilities despite having lower clock speeds than CPUs; each GPU includes 2,688 CUDA cores running at 732 MHz, allowing for faster overall throughput. CPUs primarily serve to assign workloads to GPUs rather than performing direct data processing.
Titan runs on the Cray Linux Environment—full Linux is available on login nodes while compute nodes operate on a streamlined variant.
Cooling is achieved through air circulation with chilled air passing through heat sinks located in the cabinets. The cooling system can handle up to 23.2 MW (6600 tons) of heat load by chilling water to 5.5 °C to cool recirculated air effectively. Due to noise levels in the machine room, prolonged exposure requires hearing protection.
Researchers utilize EVEREST (Exploratory Visualization Environment for Research and Technology), which includes a large visualization space featuring a main screen measuring 10 by 3 meters and an additional display capable of stereoscopic 3D and high-resolution output (37 and 33 megapixels respectively).
Projects
In 2009, the Oak Ridge Leadership Computing Facility designated six "vanguard" applications to be among the first to run on Titan due to their scientific importance and ability to maximize system utilization. These projects include:
- S3D, which models molecular combustion physics aimed at enhancing efficiency in diesel and biofuel engines. In that year, it achieved the first fully resolved simulation of autoigniting hydrocarbon flames relevant to direct injection diesel technology.
- WL-LSMS, simulating electron-atom interactions within magnetic materials across various temperatures. A previous iteration was notable for being the first application to exceed one petaFLOPS on Jaguar.
- Denovo, which models nuclear reactions to improve reactor efficiency and minimize waste production. On Titan, Denovo runs at speeds 3.5 times faster than on Jaguar and twice as fast as traditional CPU-only systems following optimization.
- Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS), a molecular dynamics code that covers scales from quantum mechanics to relativistic phenomena, supporting advancements in materials science including semiconductors, biomolecules, and polymers.
- CAM-SE, integrating the Community Atmosphere Model with High Order Method Modeling Environment components to enhance precision in global climate simulations.
- Non-Equilibrium Radiation Diffusion (NRDF), which simulates neutrino dynamics in supernovae with applications extending to laser fusion, fluid mechanics, medical imaging technologies, nuclear reactors, energy storage solutions, and combustion processes. Its Chimera code simultaneously models hydrodynamics, radiation transport, and nuclear burning.
Additional significant projects include:
- Bonsai, a gravitational tree code designed for n-body simulations; it was nominated for the 2014 Gordon Bell prize after simulating the Milky Way star-by-star at a sustained speed of 24.773 petaFLOPS.
- VERA, developed by the Consortium for Advanced Simulation of Light Water Reactors (CASL), enables detailed tracking of reactor core behavior throughout its operational lifetime. Although not part of the original six applications, VERA was adapted for Titan’s hybrid architecture with assistance from CAAR and TitanDev teams. Its objective is to simulate an entire fuel cycle—typically lasting between 18 and 36 months—in just one week.
By 2013, thirty-one applications were planned for execution on Titan, usually running four or five simultaneously.
Code adaptations
Numerous initiatives necessitated adjustments in their codebases to fully exploit Titan's GPU capabilities, while ensuring the software remained operable on CPU-only platforms to prevent exclusive reliance on Titan. The Oak Ridge Leadership Computing Facility (OLCF) created the Center for Accelerated Application Readiness (CAAR) to facilitate this transition, offering developer workshops at Nvidia's headquarters that emphasized Titan’s hardware design, compiler technologies, and application development. CAAR worked closely with Nvidia and software providers to embed GPU directives into standard programming languages, allowing scientists to express parallelism in familiar languages such as Fortran, C, and C++ for GPU execution. Computational astrophysicist Dr. Bronson Messer highlighted that achieving peak performance on Titan demands maintaining high GPU utilization despite GPUs being less versatile than CPUs. The Moab Cluster Suite enhanced job scheduling efficiency, raising resource utilization from 70% to approximately 95% in evaluated applications. Some projects experienced performance gains even when running on CPU-only systems; for instance, Denovo achieved twice the efficiency on CPUs alone.
The degree of code modification differed among projects. Dr. Messer from NRDF indicated that only a minor portion of his code executes on GPUs due to the straightforward and highly parallel nature of the computations involved. The NRDF application is developed using CUDA Fortran, which integrates Fortran programming with CUDA extensions tailored for GPUs. The initial GPU adaptation within Chimera involved its nuclear burning simulation module, with plans to progressively port additional components. While operating on Jaguar, the project simulated 14 to 15 nuclear species; Messer anticipated expanding this number to 200 species in future iterations, thereby improving the fidelity of simulations against experimental observations.