In previous posts we used two games to build intuition about computer architectures. In The Minecraft Example we contrasted client-server (a single authoritative server that owns all world state) with distributed systems (work and state spread across many machines with no single point of failure). In The Fortnite Example we went deeper into distributed systems — how Fortnite’s backend is actually dozens of independent services (matchmaking, inventory, authentication, CDN) coordinating over the network, each failing and scaling independently.
Both examples were about systems built to serve millions of users simultaneously. Now we turn to a different kind of computing problem: what happens when the goal is not to serve many users, but to solve one very hard problem as fast as possible?
What is High Performance Computing?
High Performance Computing (HPC) refers to the use of many computing resources — processors, memory, storage, and network — working in tight coordination to solve problems that are too large or too slow for a single machine.
The classic HPC workload is a scientific or engineering simulation:
- Modeling protein folding to discover new drugs
- Simulating airflow over an aircraft wing
- Forecasting weather at high resolution
- Running quantum chemistry calculations
- Training large machine learning models
These workloads share a common trait: they require enormous amounts of arithmetic, often on large datasets, and they need an answer in hours or days rather than months. The only way to achieve that speed is to break the problem into pieces and compute many pieces simultaneously — parallel computation.
An HPC system is typically a cluster: a collection of nodes (individual computers) connected by a high-speed network, managed by a job scheduler (like SLURM, which we covered in a previous post). A researcher submits a job — a script that describes what to run and how many cores and how much memory it needs — and the scheduler allocates resources and runs it.
HPC and Distributed Systems: The Same Thing?
Not exactly — but they are related. The answer depends on what definition of “distributed system” you use.
In the broadest sense, any system where multiple computers work together over a network is a distributed system. By that definition, an HPC cluster is absolutely a distributed system: it is hundreds or thousands of machines communicating over a network to collectively solve a problem.
But the term “distributed systems” in computer science usually carries more specific connotations — systems like Fortnite’s backend, Google’s infrastructure, or a cloud database. Those systems are designed around the CAP theorem trade-offs (consistency, availability, partition tolerance), fault tolerance (the system keeps running when nodes fail), and loose coupling (services are independent and communicate asynchronously).
HPC clusters look quite different along those axes.
HPC as a Subclass of Distributed Computing
The cleaner way to think about it: HPC is a specialized subclass of distributed computing, optimized for throughput rather than availability.
| Property | General distributed systems | HPC clusters |
|---|---|---|
| Primary goal | Availability and responsiveness | Computational throughput |
| Coupling | Loose (services talk via APIs/queues) | Tight (processes synchronize at every step) |
| Fault tolerance | High — designed to survive node failures | Low — a failed node often kills the whole job |
| Network | Internet-grade (TCP/IP, millisecond latency) | High-speed interconnects (InfiniBand, sub-microsecond latency) |
| Workload type | Many short, independent requests | One large job running for hours or days |
| Programming model | Microservices, REST, message queues | MPI, OpenMP, parallel I/O |
| Storage | Cloud object stores, distributed databases | Shared parallel filesystem (Lustre, GPFS) |
The key differences:
Fault tolerance is de-prioritized. In Fortnite’s backend, if one server goes down, the store service degrades gracefully and everything else keeps running. In an HPC cluster, if a node fails mid-job, the entire simulation typically crashes and must be restarted (or restored from a checkpoint). HPC clusters trade fault tolerance for raw performance.
Coupling is tight. Distributed systems services communicate asynchronously — a message is sent and the sender moves on. HPC jobs often use MPI (Message Passing Interface), where processes synchronize at barriers: every process in the simulation must reach the same point before any of them can proceed. This tight synchronization is what allows a physical simulation to stay consistent, but it also means the slowest node dictates the speed of the whole job.
The network is specialized. General distributed systems run over standard Ethernet and TCP/IP. HPC clusters typically use InfiniBand or other high-speed fabrics that offer latencies measured in microseconds and bandwidths in hundreds of gigabits per second. This matters because MPI communication between thousands of processes happens constantly throughout a simulation.
So Is HPC a Type of Distributed System?
Yes — with an asterisk.
HPC is a subclass of distributed computing in the sense that multiple computers are working together over a network. But it is a specialization that makes different trade-offs than what most people mean when they say “distributed systems”:
- It prioritizes throughput over availability
- It is tightly coupled rather than loosely coupled
- It is optimized for batch workloads rather than always-on services
- It uses specialized hardware and software not found in typical cloud infrastructure
Think of it like this: all rectangles are parallelograms, but not all parallelograms are rectangles. All HPC clusters are distributed systems, but not all distributed systems are HPC clusters. The Fortnite backend and a climate model running on 10,000 cores are both distributed systems — they just solve completely different classes of problems with completely different design priorities.
What Comes Next
HPC clusters are where SLURM lives. The job scheduler exists precisely because many researchers want to share the same cluster at the same time, and something has to decide who gets which nodes, for how long, and in what order. In upcoming posts we will look at how to write jobs that take advantage of HPC resources effectively — parallel jobs, GPU workloads, job arrays, and how to stop wasting allocation time.