Jump to content

Lustre (file system)

From Alfawiki
Lustre
Company typeOpen source software
IndustryFile system software
Founded1999
FounderPeter J. Braam
Area served
Worldwide
ProductsLustre file system
OwnerCommunity (via OpenSFS and EOFS)
Websitelustre.org

Lustre is a parallel distributed file system designed mainly for extensive cluster computing environments. Its name is derived from the fusion of Linux and cluster. Distributed under the GNU General Public License version 2, Lustre provides high-performance file system solutions suitable for clusters ranging from small groups to vast multi-site infrastructures. Since 2005, it has been implemented by more than half of the top ten and over 60 of the top 100 fastest supercomputers globally, including the leading TOP500 supercomputer as of November 2022, Frontier, alongside former top performers such as Fugaku, Titan, and Sequoia.

Lustre file systems support scaling across multiple clusters, accommodating tens of thousands of client nodes and hundreds of petabytes of storage spread over numerous servers, achieving aggregate I/O throughput measured in tens of terabytes per second. This extensive scalability positions Lustre as a favored choice for organizations operating large data centers in fields like meteorology, simulation, artificial intelligence and machine learning, oil and gas exploration, life sciences, rich media, and financial services. The system’s I/O capabilities play a critical role in these industries and continue to draw considerable attention for advanced high-performance data management needs.

History

The inception of the Lustre file system dates back to 1999 as a research initiative led by Peter J. Braam at Carnegie Mellon University (CMU). Braam established Cluster File Systems in 2001, building upon work related to the InterMezzo file system within CMU’s Coda project. The development of Lustre was supported by the Accelerated Strategic Computing Initiative Path Forward project, funded by the United States Department of Energy with collaboration from partners such as Hewlett-Packard and Intel.

In September 2007, Sun Microsystems acquired the assets and intellectual property of Cluster File Systems, incorporating Lustre into Sun's portfolio for high-performance computing hardware with intentions to integrate it alongside Sun's ZFS file system and Solaris. After Braam left Sun in 2008, Eric Barton and Andreas Dilger assumed leadership of the project. Following Oracle's acquisition of Sun in 2010, Lustre came under Oracle’s management; however, in December 2010 Oracle announced it would halt development on Lustre 2.x and place Lustre 1.8 into maintenance-only mode, raising concerns about its ongoing viability.

This situation prompted the formation of various organizations dedicated to supporting Lustre’s open community-driven development model, including Whamcloud, Open Scalable File Systems, Inc. (OpenSFS), and EUROPEAN Open File Systems (EOFS). By late 2010, most developers had departed Oracle: Braam and colleagues joined hardware-centric Xyratex after it acquired ClusterStor assets, while Barton, Dilger, and others founded Whamcloud to continue advancing Lustre software.

In August 2011, OpenSFS contracted Whamcloud to develop new Lustre features such as enhanced single-server metadata performance scaling, online distributed filesystem checking (LFSCK), and Distributed Namespace Environment (DNE) enabling multi-server metadata distribution. Concurrently, progress was made on ZFS-backed object storage at Lawrence Livermore National Laboratory. These advancements were incorporated into community releases spanning Lustre versions 2.2 through 2.4. In November 2011, Whamcloud secured a contract to maintain the Lustre 2.x source code base for ongoing testing and bug resolution during feature development.

After winning the DOE FastForward contract aimed at preparing Lustre for exascale computing systems projected around 2018, Whamcloud was acquired by Intel in July 2012. Consequently, OpenSFS transferred Lustre development contracts to Intel. In February 2013, Xyratex purchased the original Lustre trademark, logo, website, and associated intellectual property from Oracle. By mid-2013, Intel broadened Lustre’s application beyond traditional HPC settings into platforms including Hadoop. OpenSFS issued calls for proposals focused on enhancing Lustre features, developing parallel file system tools, reducing technical debt, and supporting incubator initiatives while launching the Lustre Community Portal to centralize documentation and assistance.

On April 8, 2014, Xyratex/Seagate returned ownership of the lustre.org domain to the user community, completing this transfer by March 2015.

In June 2018, DataDirect Networks (DDN) acquired Intel’s Lustre team and assets, establishing an independent division under the revived Whamcloud brand. At SC19 in November 2019, OpenSFS and EOFS jointly received ownership rights to the Lustre trademark from Seagate, formalizing community governance over the brand.

Release history

The Lustre file system saw its initial production deployment in March 2003 on the MCR Linux Cluster at the Lawrence Livermore National Laboratory, which was then ranked as the third-largest supercomputer worldwide.

The debut of Lustre 1.0.0 occurred in December 2003, providing foundational filesystem capabilities such as server failover and recovery mechanisms. Subsequent versions introduced significant improvements: Lustre 1.2.0 (March 2004) brought support for Linux kernel 2.6 and enabled client-side write-back caching; version 1.4.0 (November 2004) incorporated InfiniBand compatibility and extent-based storage within ldiskfs; and 1.6.0 (April 2007) allowed dynamic addition of object storage targets (OSTs) alongside enhancements in distributed lock manager scalability.

Released in May 2009, Lustre 1.8.0 enhanced recovery processes, introduced OST Pools to manage heterogeneous storage environments, and functioned as a transitional version compatible with both Lustre 1.6 and 2.0 clients.

Lustre 2.0 arrived in August 2010 featuring a substantial architectural overhaul that necessitated updated clients but permitted on-disk format upgrades without requiring reformatting.

Following Oracle’s development hiatus, the community-driven Lustre 2.1 (September 2011) added support for Red Hat Linux 6, expanded ext4-based OST capacity from 24 TB to 128 TB, and delivered multiple performance and stability refinements.

In March 2012, Lustre 2.2 improved metadata handling by enabling parallel directory operations and increased the maximum stripe count to as many as 2000 OSTs per file.

Lustre 2.3, released in October 2012, enhanced metadata server scalability for multi-core processors, introduced early ZFS support for object storage, and improved integration with batch schedulers such as SLURM.

Funded by OpenSFS, Lustre 2.4 (May 2013) implemented Distributed Namespace Environment (DNE) for scalable horizontal metadata growth, broadened ZFS support across MDT and OST components, launched the Network Request Scheduler (NRS) to optimize client request handling, and extended compatibility up to Linux kernel version 3.6.

October 2013 saw the release of Lustre 2.5, which added Hierarchical Storage Management (HSM) capabilities to enable enterprise-level tiered storage solutions; this remains the actively maintained OpenSFS branch with updates continuing through version 2.5.3.

Focusing on local consistency verification, Lustre 2.6 (July 2014) introduced LFSCK tools, deployed NRS Token Bucket Filter policies for quality of service control, enhanced single-client input/output throughput, and previewed striped directories under DNE for improved scalability.

Lustre 2.7 (March 2015) expanded LFSCK functionality to validate DNE consistency across multiple MDTs, implemented dynamic LNet network configuration capabilities, and added UID/GID mapping support for clients operating across multiple domains.

With the March 2016 release of Lustre 2.8, DNE striped directories were completed with migration features; security was strengthened through SELinux client integration, Kerberos authentication, RPC encryption; and LFSCK performance was further optimized.

December 2016’s Lustre 2.9 introduced Shared Secret Key authentication via GSSAPI, Nodemap client grouping for UID/GID mapping, subdirectory mounting options, support for RPC sizes up to 16 MiB, I/O hinting through ladvise, and refined management of OST pools.

Lustre 2.10 (July 2017) brought LNet Multi-Rail bonding to boost aggregate I/O bandwidth across InfiniBand, Omni-Path, and Ethernet interfaces; incorporated composite file layouts including Progressive File Layouts for flexible data striping; enhanced NRS Token Bucket Filter scheduling policies; and provided tools to manage ZFS snapshots as Lustre mountpoints.

April 2018’s Lustre 2.11 featured File Level Redundancy via mirrored layouts to raise availability and concurrent read efficiency; Data-on-MDT storing small files on flash-based MDTs to reduce latency; LNet Dynamic Discovery for automatic Multi-Rail network configuration; and LDLM Lock Ahead to minimize lock contention during concurrent writes.

In December 2018, Lustre 2.12 advanced FLR and DoM feature stability and performance; introduced LNet Network Health monitoring to improve Multi-Rail fault tolerance; implemented Lazy Size on MDT for efficient file size estimation aiding policy engines; enabled manual restriping of directories across MDTs to balance workloads; and integrated end-to-end SCSI T10-PI data checksums from client through disk.

The December 2019 release of Lustre 2.13 incorporated Persistent Client Cache utilizing NVMe/NVRAM on clients while preserving global namespace coherence; OST Overstriping to enhance shared file performance on high-speed OSS hardware; improved Multi-Rail Network Health with RDMA router support; Self-Extending Layouts allowing dynamic sizing of file components optimized for mixed flash/disk OST setups; alongside directory scaling enhancements and bulk RPC sizes expanded up to 64 MB.

Lustre 2.14 (February 2021) introduced client-side data encryption using fscrypt prior to network transmission and storage on OST/MDT devices; extended quota enforcement mechanisms to OST pools; and automated DNE directory restriping based on administrator-defined thresholds.

In June 2022, Lustre 2.15 expanded encryption capabilities to include client directory name protection; automated DNE MDT space balancing for evenly distributed metadata workloads; enabled zero-copy RDMA I/O directly into NVIDIA GPU memory via GPU Direct Storage technology; and launched User Defined Selection Policy support for nodes with multiple network interfaces.

The latest release, Lustre 2.16 (November 2024), supports extensive network addressing including IPv6 and Infiniband GUIDs beyond IPv4 limitations; enhances hybrid direct I/O efficiency by lowering client page cache overhead; and optimizes directory traversal through batched statahead prefetching of file attributes from one or more MDS(es).

This chronological overview highlights Lustre’s ongoing advancement driven by collaborative community efforts and enterprise demands—focusing on scalability improvements, performance tuning, security fortifications, and seamless integration with cutting-edge technologies vital to high-performance computing environments.

Architecture

A Lustre file system is composed of three primary elements:

  • One or more metadata servers (MDS) which manage metadata target (MDT) devices responsible for storing namespace metadata such as filenames, directories, permissions, and file organization. Unlike block-oriented distributed filesystems like GPFS and PanFS, Lustre’s metadata server focuses solely on pathname resolution and permission verification, thus avoiding input/output bottlenecks by not handling file data operations. Since version 2.4, Lustre supports multiple MDTs within a single filesystem, allowing directory subtrees to reside on secondary MDTs; from version 2.7 onward, large directories can be partitioned across multiple MDTs.
  • One or more object storage servers (OSS) that oversee file contents stored on object storage target (OST) devices. Typically, an OSS manages between 2 and 8 OSTs, each backed by its own local disk-based filesystem. The total capacity of a Lustre filesystem is the sum of all OST capacities.
  • Clients that access data through a unified namespace adhering to standard POSIX semantics, enabling simultaneous and coherent read/write operations.

In operational deployments, MDTs, OSTs, and clients run on distinct nodes interconnected via networks. Each MDT and OST is dedicated to a single filesystem, although multiple MDTs or OSTs on one node may serve different filesystems. The Lustre Network (LNet) supports diverse interconnect technologies including native InfiniBand, Omni-Path, RoCE, iWARP through OFED, TCP/IP over Ethernet, as well as proprietary networks like Cray Gemini. Support for Myrinet and Quadrics was discontinued following Lustre 2.8. To boost performance and reduce CPU usage, Lustre employs remote direct memory access (RDMA).

Backing filesystems used by MDTs and OSTs commonly utilize hardware RAID but also support ZFS and ext4. Since Lustre 2.4, JBOD configurations are possible, enabling advanced features such as compression and checksums via ZFS integration. Clients do not have direct access to the physical storage devices, preserving filesystem integrity against faulty or malicious client behavior.

OSTs present byte-range interfaces for file data access with extent locks to maintain consistency. MDTs hold inodes, directories, POSIX and extended attributes, enforce ACLs, and provide file layout details. Both components rely on enhanced ext4 ("ldiskfs") or ZFS/DMU backends implemented through the open source ZFS-on-Linux project.

Clients mount the Lustre filesystem using a Linux kernel VFS driver that connects to the servers. During mounting, clients receive a File Identifier (FID) corresponding to the root directory. Accessing a file involves a filename lookup on the MDS; after permission validation, the client obtains the file layout or creates a new file entry. For input/output operations, clients use the logical object volume (LOV) layer to translate file offsets into object locations, acquire locks for the relevant file segments, then execute parallel read/write requests directly with OSS nodes storing the data objects. This architecture eliminates bottlenecks between clients and OSS nodes, allowing bandwidth to scale almost linearly with the number of OSTs.

Following initial retrieval of file layout information, the MDS is typically bypassed during subsequent file I/O since OSTs internally handle block allocation and data management. Clients delegate all modifications to OSS nodes rather than interacting directly with storage devices, which enhances scalability for large-scale clusters and supercomputers while improving security and reliability. This approach contrasts with shared block-based filesystems such as GPFS and OCFS, which require SAN-attached clients and are vulnerable to corruption caused by malfunctioning clients.

Implementation

In a standard Lustre deployment on a Linux client, the Lustre filesystem driver is integrated into the kernel, allowing the filesystem to be mounted similarly to other local or network filesystems. Applications on the client experience a cohesive filesystem environment, regardless of whether it spans thousands of servers and multiple MDT/OST filesystems.

Within certain MPP configurations, compute nodes direct their I/O operations through a dedicated I/O node acting as a Lustre client. This approach is exemplified by the Blue Gene system at Lawrence Livermore National Laboratory.

Earlier releases of Lustre employed the liblustre software library on platforms such as Cray XT3 running the Catamount OS. This setup allowed userspace programs to access the filesystem directly. By using liblustre, computational processors could mount Lustre without requiring the service node itself to be a Linux client, enabling direct data transfers between application space and Lustre OSSs. However, this capability was discontinued following Lustre version 2.7.0 due to lack of maintenance and testing difficulties.

With the arrival of Linux Kernel 4.18, the partially completed Lustre client port was removed from the kernel’s staging area to facilitate more efficient development and porting efforts. Nonetheless, out-of-tree Lustre clients and servers continue to be supported on distributions such as RHEL, SLES, and Ubuntu, as well as on vanilla kernels.

Data objects and file striping

Lustre employs inodes located on MDTs that reference one or more OST objects instead of traditional data blocks. These objects correspond to files stored on OSTs. When a client opens a file, it obtains object identifiers and layout information from the MDS, which permits direct parallel I/O with OSS nodes without further interaction with the MDS, thereby minimizing contention.

If an MDT inode points to a single OST object, that object contains all the file’s data. When multiple objects are linked, data is distributed in "stripes" across OSTs in a round-robin fashion similar to RAID 0, commonly using chunk sizes of 1 MB or larger. This striping method boosts performance for high-throughput access to large files by scaling both capacity and aggregate I/O bandwidth in proportion to the number of OSTs involved. Additionally, independent locking mechanisms per OST enhance locking scalability for file I/O.

Each file can define its own layout parameters—including stripe count, stripe size, and OST selection—enabling tailored tuning for performance and capacity needs. For workloads involving many threads accessing distinct files, single stripes per file maximize parallelism; conversely, multiple stripes across OSTs optimize performance for concurrent access to large files.

The introduction of composite layouts in Lustre 2.10 allows different layout settings within separate regions of a single file. The Progressive File Layout (PFL) feature utilizes this capability to enhance I/O efficiency across diverse workloads while simplifying management—for instance, small files can be stored with single stripes on flash storage to minimize overhead, whereas larger files benefit from multiple stripes that improve bandwidth and load distribution.

Lustre 2.11 expanded composite layouts by adding File Level Redundancy (FLR), which provides RAID 0+1 style redundancy along with improved read performance through overlapping layouts. This release also introduced Data-on-Metadata (DoM), which stores the initial PFL component directly on the MDT inode. This reduces overhead for small files both in terms of storage space and network traffic—particularly advantageous when MDTs utilize SSDs while OSTs rely on traditional disks.

In version 2.13, OST Overstriping was implemented to allow multiple stripes on a single OST, enhancing locking parallelism. Additionally, the Self-Extending Layout feature permits dynamic resizing of layout components during write operations to accommodate individual OST space exhaustion before reaching overall filesystem capacity limits.

Metadata Objects and DNE Remote or Striped Directories

Upon mounting a filesystem, the client obtains the root directory's 128-bit Lustre File Identifier (FID). For resolving filenames, the client uses the FID Location Database (FLDB) to associate the parent directory's FID sequence number with a designated MDT, then contacts the MDS responsible for that MDT. The MDS responds with the FID of the requested entry along with a Distributed Lock Manager (DLM) lock. Operations on directories without striping are confined to a single MDT, which helps minimize contention.

In the case of DNE striped directories, the parent directory's layout includes a hashing function and a collection of MDT directory FIDs that distribute directory entries across multiple MDTs. The client's Logical Metadata Volume (LMV) applies this hash function to filenames, directing them to particular MDT shards that manage operations similarly to non-striped directories. For readdir(), results from each shard are combined and sorted by hash value, allowing efficient directory traversal using a single 64-bit cookie.

Starting with Lustre 2.15, LMV introduces round-robin and space-balanced directory layouts to enhance utilization of several MDTs. Newly created subdirectories near the root are assigned as remote directories on sequentially chosen MDTs to evenly distribute load and available space. When free space disparities among MDTs exceed 5%, directory creation favors MDTs with greater free capacity to restore balance.

Locking

Lustre employs a distributed lock manager (LDLM), modeled after OpenVMS, to maintain data and metadata consistency with comprehensive cache coherence across clients. Metadata locks are controlled by the MDT that holds the file’s inode, utilizing the FID as the lock resource identifier. These locks encompass file lookup attributes (such as ownership, permissions, and ACLs), inode state details (including directory size, contents, and timestamps), file layout (striping introduced in Lustre 2.4), and extended attributes (xattrs since version 2.5). Clients acquire multiple metadata lock bits per inode via single RPC calls but only receive read locks; write locks remain with MDS nodes to prevent resource conflicts.

File data locking is managed by OSTs where file objects are striped, using byte-range extent locks. Clients may hold overlapping read locks or non-overlapping write locks on distinct file segments, facilitating simultaneous read/write operations without I/O bottlenecks. Linux clients typically request locks in multiples of page size (commonly 4096 bytes). OSTs might grant larger extents than requested to reduce lock acquisition frequency based on current lock status and conflicts. Locks use the object’s Lustre FID as their resource identifier. Because extent lock servers scale proportionally with OST count, locking performance improves alongside filesystem size and striping complexity.

Networking

Lustre clients and servers communicate via Lustre Networking (LNet), initially built on the Sandia Portals API. Disk storage connects to Lustre MDS and OSS nodes through direct-attached storage interfaces such as NVMe-oF, SAS, FC, or iSCSI—or via conventional SAN technologies—independent from client-server network connections.

LNet supports multiple network types including InfiniBand and Ethernet (using TCP port 988), allowing concurrent availability of several networks with routing capabilities. It utilizes RDMA for data and metadata transfers over networks like InfiniBand OFED Verbs, RoCE, Omni-Path, as well as proprietary high-speed interconnects such as Cray Aries/Gemini, Atos BullSequana eXascale Interconnect (BXI), and AWS Elastic Fabric Adapter (EFA). High availability mechanisms enable seamless failover and recovery through standby servers.

With Lustre 2.10, LNet introduced Multi-Rail (MR) functionality that aggregates multiple network interfaces between clients and servers to increase bandwidth capacity. These interfaces can be heterogeneous in network type. Version 2.12 enhanced MR’s fault tolerance when multiple interfaces are present.

LNet achieves end-to-end throughput exceeding 100 MB/s over Gigabit Ethernet connections, reaching up to 11 GB/s on InfiniBand EDR links and surpassing 11 GB/s across 100 Gigabit Ethernet interfaces.

High availability

The Lustre file system ensures high availability through effective failover and recovery mechanisms, rendering server outages and restarts invisible to users. Compatibility between minor versions permits server upgrades by switching operations to standby servers, upgrading offline, and then rebooting without disrupting ongoing processes.

Lustre Metadata Servers (MDS) are configured either as active/passive pairs managing a single MDT or as active/active pairs using Distributed Namespace Environment (DNE) to serve multiple MDTs. Object Storage Servers (OSS) generally operate in active/active mode, each exporting distinct OSTs to provide redundancy without incurring additional overhead. In filesystems with a single MDT, standby MDS nodes frequently double as Metadata Server Groups (MGS) or monitoring nodes, thereby maximizing cluster resource efficiency.

HSM (Hierarchical Storage Management)

Within a unified filesystem namespace, Lustre supports multiple storage tiers, facilitating traditional HSM capabilities that archive data from primary storage to secondary layers—often tape systems complemented by disk caches. When files are archived, they can be released from the main filesystem, leaving behind stubs that initiate restore operations upon access.

Different OST types, such as HDDs and SSDs, can be organized into named storage pools selectable via file layout configurations. File migration between tiers can be conducted manually or governed automatically by the Policy Engine. Since version 2.11, Lustre enables file mirroring across OST pools using File-Level Replication (FLR) layouts to pre-load data onto faster storage tiers for computational tasks.

The primary components coordinating the integration between primary and archive storage include:

  • Coordinator: orchestrates archive and restore operations.
  • Agent: executes copytools responsible for transferring data between storage tiers.
  • Copytool: handles data movement and metadata synchronization; compatible with various archiving solutions such as POSIX-compliant systems, HPSS, IBM Tivoli Storage Manager (TSM), Amazon S3, and Google Drive.
  • Policy Engine: observes filesystem activity and enforces policies related to archiving, releasing, migrating, purging, or deleting files. The RobinHood engine is the most prevalent choice for this role.

File states defined by HSM encompass Exist (partial copy present), Archive (complete archive stored), Dirty (primary copy modified), Released (only stub remains), Lost (archive missing), No Release (file retained in filesystem), and No Archive (excluded from archiving).

Deployments

Lustre enjoys extensive adoption in numerous TOP500 supercomputers and large-scale multi-cluster environments. Over 60 of the top 100 and six of the top 10 supercomputers employ Lustre-based filesystems. Significant installations include the 700PB capacity with 13 TB/s throughput Orion filesystem powering the Frontier supercomputer at Oak Ridge National Laboratory (ORNL); Fugaku and K Computer at the RIKEN Advanced Institute for Computational Science; Tianhe-1A at China's National Supercomputing Center in Tianjin; LUMI hosted by CSC; as well as ORNL's Jaguar and Titan. Other prominent deployments include Blue Waters at the University of Illinois, along with Sequoia and Blue Gene/L systems at Lawrence Livermore National Laboratory (LLNL).

Large-scale Lustre filesystems are also operational at facilities such as the National Energy Research Scientific Computing Center, Pacific Northwest National Laboratory, Texas Advanced Computing Center, Brazil’s National Laboratory of Scientific Computing, and NASA in North America; Tokyo Institute of Technology in Asia; and Europe's CEA, among others.

Commercial technical assistance

Commercial technical support for Lustre is commonly provided as part of computing platforms or storage devices offered by various vendors. Prominent companies that have supplied such support include Hewlett-Packard (notably through HP StorageWorks Scalable File Share from 2004 to 2008), ATOS, and Fujitsu. Vendors specializing in storage hardware that incorporate Lustre support services include Hitachi Data Systems (since 2012), DataDirect Networks (DDN), Aeon Computing, among others. Additionally, some providers focus exclusively on software-based Lustre support, such as Whamcloud.

To streamline deployment and decrease expenses, cloud service providers have launched managed Lustre offerings. For example, Amazon Web Services delivers Amazon FSx for Lustre, a fully managed cloud service designed to facilitate high-performance file system operations. Likewise, Microsoft Azure features Azure Managed Lustre (AMLFS), a scalable pay-as-you-go file system tailored for high-performance computing (HPC) and artificial intelligence applications.

See also

External links

Documentation

Information wikis

Community foundations

Hardware/software vendors

Retrieved from "https://www.alfawiki.org/index.php?title=Lustre_(file_system)&oldid=26277"