icaoberg / What is Lustre?

Definition

Lustre is an open-source, distributed parallel file system designed for high-performance computing. It presents a single, shared, POSIX-compliant namespace to thousands of client nodes at once, while spreading the actual data across many storage servers so that aggregate throughput scales with the hardware rather than being capped by any one machine [1], [2].

The defining architectural idea behind Lustre is the separation of metadata from data [3]. The bookkeeping about files—their names, permissions, ownership, timestamps, and layout—is handled by dedicated metadata servers, while the file contents live on separate object storage servers. Because these two paths are independent, small and numerous metadata operations never contend with high-bandwidth bulk data transfer, and each can scale on its own.

Formally, a Lustre file system is assembled from a handful of component types:

A Metadata Server (MDS) and its backing Metadata Target (MDT), which own the namespace—file creation, deletion, lookup, and permission checks—and record where each file’s data is located.

One or more Object Storage Servers (OSS), each managing one or more Object Storage Targets (OST), which hold the striped contents of files.

Clients (compute nodes) that mount the file system and see an ordinary directory tree, issuing standard open, read, and write calls with no special code.

TL;DR

Imagine a library so large that no single librarian could ever find your book fast enough. So the library splits the job in two. One desk—the catalog desk—knows only where every book lives: which shelf, which room, which building. It never touches the books themselves. The other staff—the shelf clerks—do nothing but fetch and store pages, and there are hundreds of them working in parallel.

When you want a file, you first ask the catalog desk: where is it, and am I allowed to read it? It hands you a map. From then on you talk directly to the shelf clerks, and because your file’s pages are spread across dozens of shelves, dozens of clerks hand you pieces at the same time. The result is enormous speed, because you are never waiting in a single line.

That is Lustre: one catalog desk (the metadata server) that stays out of the way, and a huge pool of shelf clerks (the object storage servers) that deliver data in parallel. Every user still sees a normal set of folders and files—the fan-out across servers is invisible.

How Lustre works

When a client opens a file, it first contacts the MDS, which validates permissions and returns the file’s layout—the list of OSTs that hold the file’s stripes and the stripe geometry. After that initial handshake, the client communicates directly with the relevant OSSes to move data, bypassing the metadata server entirely. This is the crux of Lustre’s scalability: the metadata path and the data path are decoupled, and the high-bandwidth data path fans out across the entire storage cluster [2], [3].

Two tunable knobs govern how a file is distributed:

• Stripe count — how many OSTs a file is spread across. A large file striped over many OSTs can be read or written by many servers and disks simultaneously.
• Stripe size — how many contiguous bytes are written to one OST before moving to the next.

Users can set striping per file or per directory (lfs setstripe), matching layout to workload. A single huge simulation output benefits from a wide stripe; a directory of many small files usually does not.

Lustre typically runs over a high-speed, low-latency network fabric—InfiniBand, Omni-Path, or high-end Ethernet—using its own network abstraction layer, LNet. Peak performance assumes this kind of dedicated interconnect; on commodity networking the advantages shrink considerably.

Pros and cons of Lustre

Pros

• Massive, scalable bandwidth. Because data is striped across many OSTs, aggregate throughput scales with the number of storage servers—hundreds of GB/s to multiple TB/s on large deployments.
• Scales to enormous capacity and client counts. A single Lustre file system can span petabytes and serve tens of thousands of clients under one namespace.
• Standard POSIX interface. Applications use ordinary open/read/write calls—no special API or code changes needed to benefit from the parallelism.
• Tunable striping. Stripe count and stripe size can be set per file or per directory, matching layout to workload.
• Open source and vendor-supported. Lustre is free and open, yet also available with commercial support and integrated appliances.
• Battle-tested at scale. It backs a large share of the TOP500 supercomputers, so its behavior at extreme scale is well understood [1].

Cons

• Weak at small-file and metadata-heavy workloads. Millions of tiny files or thousands of ranks pounding one directory bottleneck on the metadata server. (Distributed namespace, DNE, mitigates but does not eliminate this.)
• Operational complexity. Standing up and maintaining MDS/MDT and OSS/OST components, tuning the network, and handling failover requires real expertise—it is not a drop-in file server.
• Requires a fast, dedicated network. Peak performance assumes a high-speed, low-latency fabric; on commodity networking the advantages shrink.
• Availability and recovery are involved. High availability depends on shared storage and failover pairs; a corrupted MDT can be disruptive and slow to recover without careful design.
• Kernel and version coupling. The client is tightly tied to specific kernel versions, which can complicate OS upgrades and patching.
• Not aimed at general-purpose or small deployments. For a handful of nodes or mixed office workloads, NFS or a local file system is simpler and often faster.

Lustre vs. other HPC file systems

Lustre is the most widely deployed parallel file system in HPC, but it is not the only option. Here is how it compares to the other systems you are most likely to encounter on a cluster.

GPFS / IBM Storage Scale (formerly Spectrum Scale)

GPFS is IBM’s parallel file system and Lustre’s closest competitor at the high end [4]. Like Lustre, it stripes data across many storage servers and presents a single POSIX namespace at massive scale. The key architectural difference is metadata handling: rather than routing all namespace operations through dedicated metadata servers, GPFS uses a distributed locking and token model in which metadata responsibility is spread across nodes. In practice this often gives GPFS an edge on metadata-heavy and mixed workloads, and it ships rich enterprise features (snapshots, tiering, integrated management, information lifecycle management). The trade-off is that GPFS is proprietary and licensed, whereas Lustre is open source and free to deploy. Sites frequently choose GPFS for its polish and support, and Lustre for its openness and raw streaming bandwidth per dollar.

BeeGFS

BeeGFS is an open-source parallel file system built for ease of deployment [5]. Architecturally it resembles Lustre—separate metadata and storage services with striping across storage targets—but it is designed to be far simpler to install and operate, and it runs entirely in user space rather than requiring a patched kernel client. It also supports distributed metadata across multiple metadata servers out of the box, which helps with metadata-intensive workloads. BeeGFS is popular for mid-sized clusters, AI/ML pipelines, and sites that want good parallel performance without Lustre’s administrative overhead. At the very largest scale, Lustre and GPFS still dominate the TOP500, but BeeGFS occupies a comfortable middle ground.

NFS

NFS (Network File System) is the classic distributed file system, but it is not a parallel file system in the Lustre sense [6]. In its common form a single server exports a directory tree to many clients, so all traffic funnels through that one server—its bandwidth and its metadata both become a ceiling. NFS is ubiquitous, trivial to set up, and perfectly adequate for home directories, shared configuration, and modest shared storage on a cluster. What it cannot do is deliver the hundreds-of-GB/s aggregate bandwidth that a large parallel job needs. Most HPC sites use both: NFS for /home and software, Lustre (or GPFS) for the high-bandwidth scratch and project space where jobs actually do their I/O. (pNFS and clustered NFS variants close some of the gap, but they are far less common in HPC than Lustre or GPFS.)

Ceph

Ceph is a distributed storage platform whose file interface, CephFS, offers a POSIX namespace over a scalable object store (RADOS) [7]. Ceph’s strength is unified storage—object (S3-compatible), block, and file from one cluster—along with strong data resilience through replication or erasure coding and a design that tolerates commodity hardware and node failures gracefully. This makes it a favorite for cloud, OpenStack, and general-purpose infrastructure. For raw HPC bandwidth on large sequential I/O, however, purpose-built parallel file systems like Lustre and GPFS still generally outperform CephFS, which is why Ceph is more common as cloud/enterprise storage than as the primary scratch file system on a top-tier supercomputer.

At a glance

Summary

Lustre is the workhorse parallel file system of high-performance computing: an open-source system that separates metadata from data so that thousands of clients can stream shared files at enormous aggregate bandwidth under a single POSIX namespace. Its dedicated metadata servers and striped object storage are exactly what make it fast on large, sequential I/O—and exactly what make it sensitive to metadata-heavy, small-file workloads. Against its peers, GPFS trades openness for enterprise polish and stronger metadata handling, BeeGFS trades some peak scale for far easier operation, NFS trades parallel bandwidth for ubiquity and simplicity, and CephFS trades top-end HPC throughput for unified, resilient, cloud-friendly storage. On most real clusters you will meet several of these at once—typically NFS for home directories and Lustre or GPFS for the scratch space where the real work happens.

References

[1] Lustre community. “Introduction to Lustre.” Lustre Documentation. Available at the Lustre documentation site.

[2] Philip Schwan. “Lustre: Building a File System for 1,000-node Clusters.” Proceedings of the 2003 Linux Symposium, 2003. PDF

[3] Peter J. Braam. “The Lustre Storage Architecture.” Cluster File Systems, Inc., 2019. arXiv:1903.01955

[4] Frank Schmuck and Roger Haskin. “GPFS: A Shared-Disk File System for Large Computing Clusters.” Proceedings of the 1st USENIX Conference on File and Storage Technologies (FAST ‘02), 2002. PDF

[5] ThinkParQ. “BeeGFS Documentation.” Available at doc.beegfs.io.

[6] Brian Pawlowski et al. “The NFS Version 4 Protocol.” Proceedings of the 2nd International System Administration and Networking Conference (SANE), 2000. RFC 7530: doi:10.17487/RFC7530

[7] Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. “Ceph: A Scalable, High-Performance Distributed File System.” Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI ‘06), 2006. PDF