icaoberg / What is a File System?

Definition

A file system is the layer of software that imposes structure on otherwise unstructured storage. A raw storage device—a hard disk, an SSD, a partition—presents itself to the operating system as nothing more than a long, flat array of fixed-size blocks, addressable only by number. That is not a useful abstraction for humans or programs. A file system is what turns that array of blocks into named, hierarchical, byte-addressable objects: files and directories.

Formally, a file system can be described as a mapping from a human-meaningful namespace onto physical storage. Let B be the set of blocks on a device. A file system defines:

A set of files, where each file is a finite sequence of bytes together with metadata (owner, permissions, timestamps, size), and a mapping M that associates each file with the subset of blocks in B that hold its contents.

A namespace, typically a tree (or, with links, a directed graph) of directories that map human-readable names to files and to other directories.

The component that ties a file’s metadata to its data blocks is the inode (index node) in Unix-style file systems [1]. An inode holds everything about a file except its name and its content: the permission bits, the ownership, the timestamps, and the pointers to the data blocks. Names live in directories, which are themselves just files whose contents are a list of (name, inode number) pairs. This separation is why a single file can have several names (hard links) and why renaming a file never touches its data.

A file system is also responsible for consistency and durability. Because a write may be interrupted by a crash or power loss at any moment, modern file systems use techniques such as journaling (recording intended changes in a log before applying them) or copy-on-write (never overwriting live data in place) so that the on-disk structure can always be recovered to a coherent state.

TL;DR

Think of a storage device as a giant warehouse full of identical, numbered bins. You can put things in bin #4,812,003, but nobody wants to remember that a photo of their dog is in bin #4,812,003 and the next three bins after it.

A file system is the warehouse’s clerk and its filing cabinet. You hand it a name—~/photos/dog.jpg—and it quietly remembers which bins hold that photo, in what order, who is allowed to look at it, and when it was last touched. When you ask for the file back by name, the clerk fetches every bin and hands you the contents in order. When you delete it, the clerk marks those bins as free for reuse.

Everything you think of as “working with files”—folders, filenames, permissions, copying, renaming—is the file system providing a friendly, named, hierarchical illusion on top of a flat sea of numbered blocks. Without it, storage is just an anonymous heap of bytes.

What a file system actually manages

A file system juggles several responsibilities at once:

1. Space allocation

It decides which blocks hold which file, and it tracks which blocks are free. Good allocation keeps a file’s blocks close together (to reduce seek time on spinning disks) and avoids fragmentation over time.

2. Naming and hierarchy

It maps human-readable paths to the underlying files and lets directories nest arbitrarily deep, giving you the familiar tree of folders.

3. Metadata

It records ownership, permissions, and timestamps for every file, and enforces access control so that one user cannot read another’s private files.

4. Consistency and recovery

Through journaling or copy-on-write, it ensures that an interrupted operation leaves the file system in a recoverable state rather than a corrupted one.

Familiar examples include ext4 and XFS on Linux, APFS on macOS, and NTFS on Windows. These are all local file systems: they manage storage attached to a single machine.

Parallel file systems

Local file systems assume one machine owns the storage. That assumption breaks down in high-performance computing (HPC), where thousands of compute nodes must read and write the same data set concurrently, at aggregate bandwidths of hundreds of gigabytes per second. No single disk—and no single server—can keep up.

A parallel file system solves this by striping data across many storage servers and letting many clients access it simultaneously. Instead of one machine funneling every request, a file is broken into chunks that are spread across dozens or hundreds of storage targets. When a client reads a large file, it pulls different stripes from different servers in parallel, so the effective bandwidth scales with the number of servers rather than being capped by any one of them. Crucially, all clients still see a single, shared, POSIX-like namespace—the parallelism is hidden behind the ordinary file-and-directory interface.

Pros and cons of parallel file systems

Pros

• Bandwidth scales with hardware. Adding more storage servers raises aggregate throughput, so the system grows with the cluster instead of hitting a single-server ceiling.
• Huge shared capacity. Petabyte-scale storage lives under one namespace that every compute node can see, which simplifies large data sets and shared results.
• Concurrent access. Thousands of clients can read and write the same file system at once, which is essential for tightly coupled parallel jobs.
• Single namespace, standard interface. Applications use ordinary files and directories; the distribution across servers is invisible to them.

Cons

• Complex to deploy and operate. Multiple server roles, dedicated networking, and failover design mean parallel file systems need specialized administration.
• Poor for small-file and metadata-heavy workloads. They are optimized for large, streaming I/O; lots of tiny files or metadata operations become the bottleneck.
• Costly infrastructure. Peak performance assumes a fast, low-latency network and dedicated storage hardware, which is expensive to build and maintain.
• Overkill outside HPC. For small clusters or general-purpose use, simpler options like NFS or a local file system are cheaper and often faster.

Lustre

Lustre is the most widely deployed parallel file system in HPC; it backs a large fraction of the world’s fastest supercomputers [3]. Its architecture separates metadata from data, which is the key idea behind its scalability [2], [4]. A Lustre file system is built from a few distinct component types:

• Metadata Server (MDS) and Metadata Target (MDT). The MDS handles all namespace operations—opening files, creating and deleting them, looking up names, checking permissions. The actual metadata lives on the MDT. Because metadata operations (which are small and numerous) are handled separately from bulk data transfer, neither bottlenecks the other.

• Object Storage Servers (OSS) and Object Storage Targets (OST). The OSSes hold the file contents. Each OSS manages one or more OSTs (the underlying block devices). A single file’s data is striped across multiple OSTs, so reading or writing that file engages many servers and many disks at once.

• Clients. Compute nodes run the Lustre client, which presents a standard POSIX file system that mounts and behaves like any other directory tree. Applications need no special code—they call open, read, and write as usual.

The flow is straightforward. When a client wants a file, it first asks the MDS: where does this file live, and am I allowed to touch it? The MDS replies with the layout—which OSTs hold which stripes. From then on, the client talks directly to the OSSes to move data, bypassing the metadata server entirely. This is what lets Lustre scale: the metadata path and the data path are independent, and the high-bandwidth data path fans out across the whole storage cluster.

The trade-off is complexity and a well-known sensitivity to metadata-heavy workloads. Lustre excels at streaming large files at enormous bandwidth, but a job that creates millions of tiny files, or that hammers a single directory from thousands of ranks at once, funnels all of that through the metadata server and can bring an otherwise fast system to a crawl. This is why HPC users are often advised to aggregate small files, avoid giant single directories, and stripe large files across many OSTs.

Pros and cons of Lustre

Pros

• Massive, scalable bandwidth. Because data is striped across many OSTs, aggregate throughput scales with the number of storage servers—hundreds of GB/s to multiple TB/s on large deployments.
• Scales to enormous capacity and client counts. A single Lustre file system can span petabytes and serve tens of thousands of clients under one namespace.
• Standard POSIX interface. Applications use ordinary open/read/write calls—no special API or code changes needed to benefit from the parallelism.
• Tunable striping. Users can control stripe count and stripe size per file or per directory, matching layout to workload.
• Open source and vendor-supported. Lustre is free and open, yet also available with commercial support and integrated appliances, so sites can choose their level of hand-holding.
• Battle-tested at scale. It runs on a large share of the TOP500 supercomputers, so its behavior at extreme scale is well understood.

Cons

• Weak at small-file and metadata-heavy workloads. Millions of tiny files or thousands of ranks pounding one directory bottleneck on the metadata server; Lustre shines with large, streaming I/O, not many small operations.
• Operational complexity. Standing up and maintaining MDS/MDT and OSS/OST components, tuning the network, and handling failover requires real expertise—it is not a drop-in file server.
• Requires a fast, dedicated network. Peak performance assumes a high-speed low-latency fabric (InfiniBand, Omni-Path, or high-end Ethernet); on commodity networking the advantages shrink.
• Availability and recovery are involved. High availability depends on shared storage and failover pairs; a failed target or a corrupted MDT can be disruptive and slow to recover without careful design.
• Not aimed at general-purpose or small deployments. For a handful of nodes or mixed office workloads, NFS or a local file system is simpler and often faster; Lustre only pays off at HPC scale.
• Kernel and version coupling. The client is tightly tied to specific kernel versions, which can complicate OS upgrades and patching.

Summary

A file system is the abstraction that turns a flat array of numbered storage blocks into named files and directories, tracking where each file’s data lives, who may access it, and how to stay consistent across crashes. Local file systems like ext4 and NTFS serve a single machine; parallel file systems like Lustre extend the same familiar interface across hundreds of servers, striping data so that thousands of clients can read and write shared data at massive aggregate bandwidth—provided the workload plays to their strengths and avoids drowning the metadata server.

References

[1] Marshall K. McKusick, William N. Joy, Samuel J. Leffler, and Robert S. Fabry. “A Fast File System for UNIX.” ACM Transactions on Computer Systems, 2(3):181–197, 1984. doi:10.1145/989.990

[2] Philip Schwan. “Lustre: Building a File System for 1,000-node Clusters.” Proceedings of the 2003 Linux Symposium, 2003. PDF

[3] Peter J. Braam. “The Lustre Storage Architecture.” Cluster File Systems, Inc., 2019. arXiv:1903.01955

[4] “Understanding Lustre Internals,” Second Edition. Oak Ridge National Laboratory / Lustre community documentation. Available at the Lustre documentation site.