icaoberg / Lustre Components: A Glossary

Overview

A Lustre file system is not a single program but a collection of cooperating services spread across many machines. Its power comes from separating metadata (the bookkeeping about files) from data (the file contents), and from striping that data across many storage servers so thousands of clients can read and write in parallel [1], [2].

The terminology can be dense, and it draws a careful distinction between servers (the machines that run a service) and targets (the block devices where the bytes actually land). This post is a plain-language glossary of the components you will meet when you use Lustre file system.

Core storage components

MGS — Management Server

The Management Server is the registry for the whole Lustre installation. It stores and serves configuration information—which targets belong to which file system, and how they are connected—to all other components as they start up. A single MGS can manage multiple Lustre file systems. It is the first service that comes up and the one every other node contacts to learn the cluster’s layout.

MGT — Management Target

The Management Target is the block device (a small disk or partition) on which the MGS stores its configuration data. It is tiny compared to data storage—configuration is only a few megabytes—but it must be present and reachable for the file system to mount.

MDS — Metadata Server

The Metadata Server owns the namespace: it handles every operation that touches file identity rather than file contents—creating and deleting files, looking up names, opening and closing files, checking permissions, and returning a file’s layout (which OSTs hold its stripes). Because metadata operations are small and extremely numerous, giving them dedicated servers keeps them from contending with bulk data transfer. A file system can have several MDSes when the namespace is distributed (see DNE).

MDT — Metadata Target

The Metadata Target is the block device where the metadata actually lives: the directory tree, inodes, filenames, permissions, and per-file layout information. One MDS serves one or more MDTs. The very first MDT (index 0) holds the root of the namespace. Metadata performance—how fast you can create, stat, and delete files—is largely a property of the MDT hardware.

OSS — Object Storage Server

The Object Storage Server is a machine that stores and serves the contents of files. Once a client has learned a file’s layout from the MDS, it talks directly to the OSSes to move data, bypassing the metadata path entirely. Aggregate bandwidth scales with the number of OSSes: add more, and the file system can move more data at once. Each OSS typically drives several OSTs.

OST — Object Storage Target

The Object Storage Target is the block device where file data physically resides. A single file is striped across one or more OSTs, so reading or writing that file engages many disks and servers simultaneously. The total capacity of a Lustre file system is the sum of its OSTs; its peak bandwidth is roughly the combined bandwidth of the OSTs a file is striped over. OSTs are the workhorses of the system.

Client

A client is any node—usually a compute node—that mounts the Lustre file system and sees a normal POSIX directory tree. The Lustre client software makes the distributed storage look like an ordinary mounted volume: applications issue standard open, read, write, and close calls with no special code. The client is what coordinates the two-step dance—ask the MDS where, then fetch data straight from the OSSes.

Networking and layout terms

LNet — Lustre Networking

LNet is Lustre’s own network abstraction layer. It provides a uniform messaging interface over many kinds of physical fabric—InfiniBand, Omni-Path, RoCE, and TCP/IP over Ethernet—and handles routing between them. LNet is what lets a client on one network reach storage servers on another, and it is tuned for the high-bandwidth, low-latency communication Lustre depends on.

Striping

Striping is the practice of splitting a file’s data into chunks and distributing those chunks across multiple OSTs. It is the mechanism behind Lustre’s parallel bandwidth: a widely striped file can be served by many OSTs at once. Striping is controlled by two parameters below.

Stripe count

The stripe count is the number of OSTs across which a file’s data is spread. A stripe count of 1 keeps a file on a single OST (fine for small files); a high stripe count spreads a large file across many OSTs for maximum parallel throughput. It is set per file or per directory with lfs setstripe.

Stripe size

The stripe size is the number of contiguous bytes written to one OST before the data wraps to the next OST in the stripe. Larger stripe sizes favor big sequential I/O; smaller ones can help certain access patterns. Together with stripe count, it defines a file’s layout.

DNE — Distributed Namespace

Distributed Namespace (DNE) lets a single Lustre file system spread its metadata across multiple MDTs rather than funneling it all through one. This relieves the metadata bottleneck for workloads with huge directories or heavy file-creation rates. DNE comes in two flavors: remote directories (a whole subtree lives on another MDT) and striped directories (a single large directory’s entries are spread across several MDTs).

llite / Lustre client stack

The client-side kernel module (historically called llite) is the piece that presents the POSIX file system to applications and translates their calls into the MDS and OSS protocol exchanges. It is tightly coupled to the kernel version, which is why upgrading the OS on a Lustre client requires a matching client build.

lfs

lfs is the Lustre user-space utility. It is how you inspect and control Lustre-specific behavior: lfs setstripe and lfs getstripe manage striping, lfs df shows per-OST usage, and lfs find searches with Lustre-aware criteria. If you use Lustre as a power user, lfs is the command you will reach for most.

How they fit together

Putting the pieces in order, a typical read looks like this:

1. The client contacts the MGS at mount time to learn the file system’s configuration.
2. To open a file, the client asks an MDS (which reads from its MDT) for permissions and the file’s layout.
3. Armed with the layout, the client talks directly to the relevant OSSes, which read the striped data from their OSTs.
4. All of this messaging travels over LNet.

The metadata path (MDS/MDT) and the data path (OSS/OST) are independent, and only the data path fans out across the whole cluster. That separation is the single idea from which every other Lustre design decision follows.

Quick reference

References

[1] Lustre community. “Introduction to Lustre” and “Understanding Lustre Internals.” Lustre Documentation. Available at the Lustre documentation site.

[2] Peter J. Braam. “The Lustre Storage Architecture.” Cluster File Systems, Inc., 2019. arXiv:1903.01955