The true cost of a GPU cluster

A single Blackwell GPU can cost more than a car. It can use more power than a home. And many AI startups spend more money on GPUs alone than on their entire human team. GPUs are expensive. For a long time, the industry focused on a single value: how much does a GPU cost per hour? We thought we got it all figured out. It's a simple and effective metric, right? But here's the strange part. The GPU itself might not be the thing that determines whether your cluster is actually cheap or expensive. Because when an AI company rents tens of thousands of GPUs or even more, the sticker price, the dollars per GPU hour, doesn't really tell you all that much. The question shouldn't be, how much does one GPU cost per hour? It should be, how much useful work do I get per dollar? And once you ask that question, the entire economics of AI infrastructure starts to look very different. Because 2 clusters can have the exact same GPU-hour price, the same number of GPUs, even the same generation of accelerators and the same headline specs, and yet, one of them can still be 10, 15, even 50% more expensive in reality. Not because the GPUs are different, but because the cluster is different, the storage is different, the network is different, the support is different, the setup time is different, and most importantly, the amount of useful training time, the goodput, is different. In this video, we're going to break down the real cost of GPU clusters, not the marketing version, not the simple price per H100, but the version that actually matters, when you're training models, running experiments, or serving inference at scale. Let's start with the obvious. When people compare GPU clouds, they usually look at one single metric: dollars per GPU hour. And these numbers matter, of course they do. If you're renting thousands of GPUs for months or even years, small differences add up very quickly. But the mistake is treating GPU-hour pricing like it's the same thing as total cost. It is not. A GPU-hour is only valuable if the GPU is doing useful work. Makes sense. If the job is stuck waiting for storage, if NCCL is hanging, if a node failed and nobody noticed for an hour, if your training run has to restart from the last checkpoint, and if engineers are spending weeks tuning the cluster, just to get expected performance, you are not getting your money's worth. You are still paying for the GPUs, but you are not getting useful work from them. It's almost like a hidden tax. And at large scale, this tax becomes brutal. Especially if you didn't plan for it, and in order to plan for it, you have to be aware of it. Modern AI clusters are not one big computer, they are thousands of separate machines stitched together by networking, storage, orchestration software, monitoring systems, health checks, support teams, and fault tolerance strategies. The GPU is the expensive part, but the system around the GPU determines whether the GPU is productive or not. So what is the real total cost of a GPU cluster? What do you have to consider? Let's break it down into 8 areas. First, the GPUs. This is the main cost: dollars per GPU hour multiplied by the number of GPUs multiplied by time. That's what everyone looks at first, and rightfully so, but it's just one of 8 areas to look at. Second is storage. Training data, checkpoints, logs, model weights, evaluation datasets— all of that has to live somewhere. And storage is not just about capacity, it's about performance. A slow storage system can make job initialization take longer, checkpointing slower, and recovery more painful. Third is networking. There's the backend network, InfiniBand, ROCE, EFA, NVLink, and so on, which determines how well GPUs communicate during training. And there is the frontend network, public IPs, firewalls, load balancers, and all the cloud networking costs people forget until the bill arrives. Fourth is the control plane. Login nodes, Slurm controllers, Kubernetes control, orchestration layers, CPU nodes for data processing and all the services that make the cluster usable to begin with. It's not free. Fifth is support. This is one of the big hidden differences between providers. Some NeoClouds include direct engineering support in the price, while hyperscalers may charge a support percentage on top of the cloud bill. And the quality of support matters a lot when a training run worth hundreds of thousands of dollars is stuck. Sixth is the goodput. This is the most important hidden cost. Goodput isn't only a pun on throughput and good, it's the amount of useful work the cluster actually performs. Throughput says, how fast can the system run when everything works? While goodput asks, how much progress did we actually make after failures, restarts, debugging, and downtime? Seventh is setup. A cluster does not magically become productive on day 1. Some clusters need days or even weeks of tuning. Some require painful work on storage, drivers, firmware, NCCL settings, EFA tuning, Slurm configuration, Kubernetes operation, health checks, and benchmarking. And while engineers are doing that work, the cluster already costs money. And last but not least, number 8 is debugging. This is an ongoing cost. The cluster works until it doesn't. A GPU falls off the bus, a collective stalls, a job hangs during checkpointing, a node has ECC issues, or a network path behaves strangely. But how do you figure out what's wrong? With debugging. All of that takes engineering time, and engineering time is not free. The real cost of a cluster is not GPU price times hours, but it's rather GPU price plus storage plus networking plus control plane plus support plus Goodput loss, plus setup, plus debugging. That's the actual bill. Let's talk a bit more about Goodput, because this is where the economics get interesting. But what is it really? Imagine you're running a huge pre-training job. I'm talking about thousands of GPUs or more. Suddenly, one GPU fails. That sounds small, it's just one GPU out of thousands, right? But distributed AI training doesn't work like a thousand independent computers, it works more like a single machine spread across Thousands of accelerators. If one rank hangs, a whole communication group can stall. If one node fails, the job may need to restart. And if the failure is detected late, you lose time. If checkpoints happen once per hour, you may lose part of that last hour of work. If job initialization takes 10 or 15 minutes, you lose that too. And if the failed node has to be repaired or replaced before the job can continue, you lose even more. Now multiply that by hundreds of interruptions per month. That is why cluster reliability gets worse as cluster size grows. Not because each individual GPU necessarily fails more often, but because the chance that something fails somewhere in the job becomes much higher. A small cluster can go a long time without a failure, but a massive cluster is constantly fighting probability. That means if you are renting GPUs, The important question is, how quickly does the provider detect failures? How quickly can they replace a failed node? Can the workload tolerate failures, or does everything restart from the last checkpoint? This is why goodput is not just a technical metric, it's also a financial metric. Bad goodput means you're buying expensive GPUs and then throwing away some of the work they produce. Being aware of this is the first step. But it doesn't change the fact that no matter how small your cluster or how good your data center team, failures always happen. There are basically 3 ways a cluster can respond to failure. The worst case is checkpoint restart with cold spares. What does that mean? Something fails, the job stops. You identify the problem, you wait for the node to be repaired or replaced. Then you restart from the last checkpoint. This can take hours or even days in bad cases. The better case is a checkpoint restart with hot spares. But what's the difference between cold and hot? With hot spares, when something fails, the job still restarts, but it can restart much faster on an idle spare node, or a node from a spare pool. The hot spare is basically a node waiting to jump in. It's like a hot spare tire ready for your F1 car if something goes wrong. The spare node is for AI data centers what the blanket over the tire is for racing cars. You still lose some time, but the recovery time is much shorter. Hot is better than cold, both for tires and for AI clusters. The third case is fault tolerance. Instead of stopping the whole job, the system keeps running or even recovers in place. For inference workloads, this is already common. If one inference node fails, the load balancer stops sending traffic to it, and requests are retried on healthy nodes. For training workloads, this is much harder, but there are emerging approaches. TorchFT is an open-source approach from the PyTorch ecosystem. AWS has checkpointless training in SageMaker HyperPod, and Clockwork has TorchPath. But each approach has trade-offs. TorchFT can reduce checkpoint dependency, but the blast radius can be the replica group and there can be performance overhead. AWS checkpointless training uses model redundancy and can recover much faster, but it introduces memory overhead. And TorchPass tries to preserve training performance by using scheduler-level migration and idle spares, but that means reserving capacity that is not normally doing productive work. Fault tolerance saves downtime, but it may cost memory, performance, idle GPUs, complexity, or software licensing. But as clusters scale beyond 1,000 GPUs, you can't ignore it anymore. The training codebase has to be designed for the reality of the cluster. Because at that scale, failures are not edge cases anymore, they're part of normal operation. Now that you're aware of the problem, you know cost per GPU hour is important, but that it's only one part of the puzzle. But unless you run your own data center, what can you really do with this information? That's where ClusterMax comes in. If you're watching a SAM analysis video on hidden GPU cluster costs, I'm pretty sure you've heard of ClusterMax before. If not, it's an independent GPU cloud ranking by SemAnalysis. And what makes ClusterMax so special is that it's not just looking at price per GPU hour, but including the whole picture, which means everything we just talked about. If you're interested in an actual deep dive into how different ClusterMax tiers stack up to each other using a holistic good-put approach, I recommend you check out the SemAnalysis article on GPU cluster cost. It's basically a much more in-depth version of what we just scratched in this video, and it also includes a detailed total cost comparison across 3 different scenarios: a large-scale LLM pre-training run, multimodal RL research, and inference endpoints. I'm not gonna say the results will shock you, but it's actually insightful to see the differences a provider can make, which directly results in very different TCO metrics. So, if you are buying GPU capacity, what should you actually ask before you commit? Don't just ask, what is your price per GPU hour? Instead, ask, what is the cluster-level mean time in between failure? How often do jobs fail? How fast do you identify a failed node? How fast do you replace it? Do you have hot spares? Who manages the spare pool? Are health checks configured by default? Who can access the monitoring dashboards? What storage system do you use? What throughput do I actually get? How long does job initialization take? Is support included? Do I get direct engineers or ticket routing? Is the proof of concept free? How long does cluster setup usually take? What CIRM or Kubernetes stack do you provide? What NCCL tuning is required? What performance benchmarks can you show for my workload? And most importantly, how much good put should I expect? Because in the end, you are not buying GPUs. You are buying time to research. You are buying training runs completed. You are buying inference tokens served. You are buying model progress. A GPU sitting idle during debugging is not cheap. A GPU waiting for a checkpoint restart is not cheap. A GPU attached to slow storage is not cheap. And a GPU in your cluster that takes weeks to tune is not cheap. The cheapest GPU is the one that helps you finish the work fastest. I hope you found this video interesting, if you did, subscribe for more deep dives into AI infrastructure, semiconductors and the hardware behind AI. And see you in the next one!