Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs)

Hello everyone. Welcome back to Semi Analysis Weekly. We are here this week with a ClusterMax 3.0 review podcast. I got Sam and Pratt. We're gonna talk through Everything that was included in the article that we pushed out on Wednesday, September 23rd. We'll talk through the rankings as they are today. Some of the statistics, like how many providers we actually tested, the methodology that we used to go through the testing over the last number of months and tease some upcoming projects such as Endpoint X, Post-Training X, Harness X, Sandbox X. We're going to test everything and then talk about some of the trends that are affecting the NeoCloud industry. financing, the move to Vera Rubin from Blackwell Security. We had a previous article on that, eugenic coding, how that's impacting infrastructure. And maybe we'll get into some individual stories about specific topics. But guys, welcome to the show. How was the last 3 months? Sam, you're coming out of your hole, man.

Yeah, it's my first time seeing sunlight today, actually, since testing began. But yeah, I mean, I'm a little sunburnt from it, but, you know, I've had a few lifetimes worth of reps on Kubernetes and Slurm. So wouldn't, wouldn't trade for anything.

Prad, how about you? You feeling good after the testing period has come to a close?

Yep, feeling pretty good. Was a lot of fun testing out these clusters. You know, always fun playing with big infra.

Awesome. So headline news, I think the biggest thing is that there's a new addition to the Platinum tier. We've got Nebula up there, Joining CoreWeave. In the gold tier, rounding it out, we've got Oracle and Google Cloud, Google joining Oracle. On the silver side, Azure moves down, Firmus and Lambda stay, GMI moves up, and TensorWave stays at silver. And then into bronze, companies like AWS, Crusoe together, all moving down. Rounding out the list at a high level, this time we rank 77 providers. We increase our market view to cover 323 total, which is available for subscribers to our NeoCloud model in great detail. And we've now interviewed well over 200 end users of NeoClouds as part of this research. So let me pose a question to Sam to start. Most interesting insight that comes to mind when you think about the labs that we've interviewed so far, like reliability, Performance, ease of use, support? What do you think is top of mind for people that are buying GPU clusters right now?

It's a good question. I mean, the first thing that comes to mind is just the ridiculous backwardation of the price curve at the moment. So people are willing to pay just about anything for any chips that they can get in, you know, the next few weeks, especially. You can get a big discount if you're willing to wait even a few months. And yeah, I mean, that means that just about everyone in the industry looks like a genius right now. If you have chips to sell, even if, you know, your 5-coded Kubernetes is completely broken, even if storage barely works, if you've got GPUs, you're going to find people willing to pay a pretty good margin to take them off your hands.

Yeah, it's interesting. We've talked about that trend a few times. previously, which is like the providers that weren't able to sell their stuff early, typically that's a signal that they're not as high quality, but now they're actually closing these deals for significantly better margins, which is interesting. Prat, how about you? We're expanding a little bit beyond ClusterMax in the future, things like serverless inference endpoints, posted, like post-training infrastructure for RL harnesses, sandbox. You've been digging in there when you're talking to the providers themselves. About the roadmap, do you find it confusing that people can't get Slurm and Kubernetes right sometimes, but they're already ready to launch like 4 new products?

I mean, I guess, right? Because currently the margins on compute are so high. Of course, as a neo-cloud, you do want to capitalize on that. You can, of course, like SpaceX provides really good compute to Google. They have deals with Anthropic and also several other Neo Labs. However, NeoCloud also wants some of that, you know, like some of that cake. So they're thinking, what is the most high-margin business that we can enter right now? Managed clusters, of course, would be one of them, but it's a huge headache setting up the entire cluster. So why don't we just go above that layer and sell actually hosted training? Why don't we sell managed inference? You know? So they're just in the end, just expanding their market, trying to earn that little like good cash, which the labs are earning right now. But everything points to centralization of compute. That's the main thing I can say, centralization of compute.

Makes sense. Sam, how about you? Like, there's a few different providers who have been on the cluster max ranking system in the past and are either moving down the list or just moving to unavailable because they're either focused on bare metal or maybe they're focused on their inference endpoint business over clusters. But I don't know, I talk to NeoLabs all the time and a lot of people still need training clusters. A lot of people host inference for themselves and they still need those clusters. Like, do you think there's no more to be done on the product roadmaps for people that are building managed cluster experiences?

It's interesting, as, as Prat said, there is an interesting fluidity to the market right now where, you know, some of the better managed cluster providers know how to manage a cluster because they have teams internally who are trying to use these things and can give good feedback for, you know, what a good Slurm on Kubernetes layer looks like, how to set up health checks, that sort of thing. So there definitely is a benefit to integrating these things in-house, in addition to just the fact that you have another service that you're offering on top where you can take margin from another layer. I mean, we certainly see plenty of room for improvement for most of these players. You know, I don't think that we're requesting super fancy bells and whistles on top of these managed clusters. You know, there's a handful of genuinely interesting additions to the managed cluster experience, but mostly I think that there's plenty of work to do just to get Slurm and Kubernetes bulletproof, to get health checks right. And I think that, Everyone on this call would agree that it's quite lucrative for these companies to do a better job, be able to collect a few more cents on top of the GPU hour prices that they're charging just by doing a slightly better job at the software layer and picking one lane and making it great.

Okay. Let's dig into health checks because this is something that we wrote a dedicated article about a little while ago called How Much Do GPUs Really Cost? And In this one, Clostrimax 3, we go through in detail the methodology that we approach testing reliability with. Can you tell me a story, leading the witness here, about some providers that claim to have health checks and then when we test them or try to exercise them with them hand in hand, they're just not really working or— they're needing a lot of manual intervention. Like, what sets apart somebody that has health checks developed out to somebody that doesn't? And maybe just basic level, define a health check, right? Sure.

So there are basically 2 types of health checks, active health checks and passive health checks. So an active health check runs an actual workload or sort of a dummy workload when there's nothing else going on in the cluster. This is normally preemptible. So if I have an active health check running and then I decide that actually I have some training job I wanna run, the health check gets interrupted and the actual workload begins. The passive health check is just running in the background. You know, you wanna see as little performance overhead as possible on the passive check. It doesn't— it can be exactly zero. It can just be, you know, reading logs from the CPU while the GPU does all its work. And I mean, there's a handful of basic things that we like to see in health checks that are not always respected. I mean, one, like, basic thing is, like, if you just diagram the health check on paper, is there any chance that it works? So, you know, there's a million corner cases that you can come up with for, like, you know, some obscure race condition that causes the health check to fail. But, you know, we've been through a few of these clusters where there's just no way it could possibly work. One example I'm thinking of, and these are, I guess, health checks that are so bad, they're worse than no health checks, because they're, like, actively interfering with jobs. On the Amazon HyperPod Slurm, the first time we tested, there was a health check that's scheduled in the auto-remediation phase that needed the node to be healthy before it could run. So this was, like, to make sure that the node had recovered its health, And there was like an obvious contradiction that it could only run once the node was back in the fleet and it was needed to bring the node back into the fleet. And there's— I think there are probably 5 or 10 examples of this throughout testing where the health check just never had a chance. And, you know, it— I don't know, it makes you wonder whether these people are really putting it through the paces before they hand it off to to customers, given that, you know, our simple little XID injection catches some basic failures of functionality.

Yeah, it makes sense. On the XID injection, maybe you can explain like a little example of the process that we go through to actually simulate a failure, because, you know, it's different on managed clusters versus endpoints, and, you know, in terms of like how things can fail or how that affects a customer workload. And when we go through the testing, I mean, we're like, for some of it we're doing custom stuff, but the basic way with DCGM Inject, it's using the basic documented approach that NVIDIA releases, right?

Yeah. It's nothing nearly as complicated as some of the other things that these companies handle in their day-to-day. So, I mean, the most basic version is just You take a node, as long as you have the requisite privileges, you write into the kernel ring buffer the message that exactly corresponds to the message that would be emitted in the event of a genuine GPU hardware failure. This doesn't always work. People have health checks configured in different ways, but normally you can like literally just read out the health check and figure out how it works and make sure that this is in fact tripping the wire that you need to trip. Most of the time, this is Exactly what you need to do. So this springs into motion the health check exactly as it would run in production. And yeah, I mean, once that happens, it's a bit of a Rube Goldberg machine sometimes. Ideally, it's like straightforwardly quarantined. There's some passive check that runs before it's auto-remediated. Everyone makes sure it's healthy. You know, depending on the exact error that you get, you want to handle it in different ways, but The sort of supported path is pretty straightforward in general, and there's not that much to say on it. It's taken out, you can't see it, it's validated, it's brought back in, and then you can schedule jobs onto it again. In the meantime, because it was unhealthy, you want to make sure that you're not able to schedule jobs on it and slow down your workload. But there's a lot that can go wrong in between the simulated failure and getting your node back into your fleet again.

Yeah, and that's exactly what we're doing, right? Simulating a failure, testing the amount of time that it takes for the provider to identify that failure has occurred. Followed by testing the amount of time it takes for them to drain the node, get it out of the cluster, replace it with a hot spare in the case of the HGX machines, or do some sort of thing like a reboot or repair workflow in the case of the NVL72 rack-scale architectures. Now, this is when it comes to hardware failures on a cluster. And I think the reason why we dig in here so much is that people are like actively paying money for these GPUs that they then can't use if they fail. And that is basically the only way you can, as a customer, have any sort of recourse with the provider in the event of these failures is if you have SLAs enshrined in a contract that you both signed up for that say you're going to get this amount of credits if you have downtime. You have to define the downtime. You have to make sure that there's not a bunch of excuses from it. And for this reason, you should be contacting SemiAnalysis for access to our sample SLAs so that you can You know, get these contracts to the industry standard for what other people do agree to with other, you know, customers. Anyway, it makes me think about people paying money for things that they're not, you know, getting in return, right? Or spending too much money, more than they have to. So, Prat, maybe you can tell us a little bit about your initial experience testing some of these providers' endpoints as a bit of a teaser. It's not included in the ClusterMax experience, but a teaser. You know, in this initial article, but it's still something that we are, you know, tracking and we're going to be writing more about soon. So maybe give us a preview on like what you've experienced so far when we said, okay, initially when we said, let's go into endpoint testing, it was like, what's the difference? It's just the API. Everybody's API is going to be the same, but no, there's some real reliability challenges and some differences in costs that you've already teased out in like a week of testing, right? Yeah.

Yep. Full disclosure, this is the preliminary test. We are going to do a huge test right after this, which I call the hero run, which is going to test concurrency up to like 1,024 or even more than that. So from the preliminary runs, something that, which is very interesting, and as Jordan, you said that a lot of people would assume that an endpoint just works. An endpoint is like a Lambda function. You just execute the function and you get an output. Like that, you know, however, a lot of providers underneath have their hardware running in such a configuration, be it auto scaling, be it their cache-aware routing is not working, that the cache hits don't like deliver really. So for one provider that we tested, and it's a very famous provider, I cannot name right now. However, for the one provider, for the one provider we tested, we got a cache hit rate of around like Now you would say 75% seems, wow, like seems decent, right? Above like 50% is fine. And for the other good providers, you'll see a cache hit rate of 99%. FYI, this workload was around, you could say, 800 tokens for a trace. So we're just doing replays on the single endpoint. But that 75% cache hit rate gave almost doubles the cost. Doubles the total cost of running this replay. And it's quite shocking. You would assume that 75% is pretty decent. 75% is fine. However, for these providers, even a 75% would double the cost. So imagine if you're paying $200, you'll be paying $400 for the same accuracy, for probably even slower speeds, because again, cache-aware routing affects speeds. And again, you'll have a huge bill for the same exact model, for the same exact accuracy. It's quite shocking.

Yeah, that's become a bit of a theme, you know, getting shocked by the— by what you're having to pay after you actually go through your time spent actually running on something, right? Okay, let's talk a little bit more about some of the technical details on these tests that you ran on a lot of these clusters, Sam. So beyond reliability and beyond pricing, let's say, there is differences in performance or performance that you're able to realize out of the box with a reasonable amount of testing. Let's start by talking about networking and then we can go into storage. When it comes to getting good performance outta the network and it comes to troubleshooting this with customers, or, you know, do you think there's specific providers that like stand out as being good at this? I mean, obviously platinum, gold tier, silver even is where the good providers are sitting in this ranking, but what stops somebody from giving us a cluster and us being able to realize full performance on a training job or a— Like microbenchmark on the network out of the box?

Yeah, it's a good question. I think that, you know, people hear about GPUs and, you know, the top line number that's normally quoted for a new generation is the flop spec. But it turns out that this is something that basically doesn't distinguish cloud providers. It's, you know, reliability is the first thing. Obviously, if you can't use the chips at all, then that's a bad deal. But networking is probably the next most important thing that we check. And I mean, basically the simplest advice we can give to providers is like, make sure there's a golden recipe for NCCL, that NCCL being the NVIDIA Communications Library that's sort of used by default on these clusters. Make sure there's a golden recipe for NCCL that gives me reasonable perf out of the box. So I don't wanna have to go hunting down, you know, strange interface configurations, you see pretty frequently that there are like pathological message sizes. So for whatever reason on this cluster, you get a huge dip at 128 kilobytes or maybe even at excessively small message sizes, it actually won't go through at all. So there's like a fairly straightforward test that you just like sweep all the message sizes, you sweep in multiples of 2 the number of nodes that you have on the cluster. You wanna see monotonic increases in throughput as the message size increases. There's a bit of detail to contend with here because actually NVIDIA's default library isn't perfect out of the box. And sometimes due to just bad software, bad heuristics by the routing, you'll get dips in performance. But basically you should see a smooth logistic curve and I shouldn't have to, you know, go into niccol.conf and rename things or, you know, figure out how exactly this interface is exposed to Kubernetes. I just want to have like, yeah, somewhere waiting for me on the cluster, something that I can hit enter and get a reasonably good starting point to hill climb on. I can tune that to my heart's content depending on what model I'm serving and what my parallelism strategy is and so on. But something that we think all providers should do is just provide a golden recipe for anyone as soon as they get on the box.

Yeah.

And we saw a big win actually out of this from Google, which is that they built this into the NVIDIA NGC containers now so that their custom networking, which does not use NVIDIA's silicon in the switches or in the network cards, is still able to work out of the box with Nickel. And they don't have to do a bunch of, you know, controls in the versioning that where they have to keep track of every single parameter that Nickel can tune in that container, how it changes over time, and then tune their network to accommodate it. It just kind of works out of the box in there. On the alternative side, I think a lot of people felt like with, you know, agentic coding, software's going to be free. And so therefore EFA from AWS is going to be easier to use. But I think what's happened is that instead of everybody sticking with Nickel and just making, you know, the defaults easier to use, which to be fair, EFA out of the box with Nickel is easier to use than the last time we tested it. What you've now seen is that collective libraries like DPP or Munip are actually built on top of NVSHMEM, a lower-level library than Nickel, to get extra performance. And this is just like completely not supported by EFA out of the box. So maybe you can talk about like the headache of, or the double-edged sword that is a provider trying to save some money by controlling the network. designing it custom in their way, while then actually having to make it work for customers out of the box.

Yeah, we've had, we've had our fair share of conversations in Slack about EFA. I mean, the bottom line is that, you know, our Inferentia team and the people that they collaborate with, and people throughout the industry are working really hard, really hard on hill climbing these performance metrics. for scale-out networking. You know, the DeepSeek Collective's libraries stand out. There's a handful of other ones that the InferenceX guys could tell you all about. And you want to be able to take advantage of all that work by the open-source community and just take their recipes and use them on your cluster, take their recipes and improve them in some way that is important to your workload. I mean, we could have Dan on the pod to explain to you basically how the bomb breaks down when you're building out a big cluster. But something like, and Jordan, you can fact-check me on this as well, something like 75 or 80% of it comes down to stuff other than networking. Like scale-out certainly matters. If you can save a few percent, it's nice. But if you save a few percent and like keep people from using modern workloads because you know, they can't use the DeepSeek communications recipe that they would like. That's absolutely not worth it. You know, we've seen some cool blog posts by people at Perplexity and elsewhere explaining how to get reasonable perf out of VFA. But I mean, yeah, you can see the money that they've saved by building their own custom scale-out is being paid instead to engineers who have to spend extra hours making sure that the workflows run reasonably well on their weird scale-out that nobody actually wants to use. So yeah, it would just be so much better if the industry's best player used a stack, hardware and software, that made use of everything else going on in the field.

Yeah. And this is not to say that like open-source software is the be-all end-all. I mean, going back to inference endpoints, maybe, Praat, you can chime in and say like, everybody wants to use VLLM and SG Lang, but what we've seen from these endpoint providers is that you actually need to build a whole bunch on top of open source just to get this stuff to work correctly. You can't just expect to plug GPUs in, turn them on, install the open source software, and then it all just like works and you have this great business. Like you need to actually do a bunch of extra work too.

Wait, no way. I thought, I thought you could just earn like 80% margins on the inference business, right? Inference endpoint business. But just kidding. Just kidding. You're right. You're right.

Maybe if your company policy is to never give anybody a discount for Getting a cache hit, then you can earn these markers, right?

Yeah.

But yeah, Jordan, you're absolutely right. A lot of people think we have this cluster. A lot of neoclouds right now think that we have this cluster. We are shit on cluster max, so might as well create a managed inference endpoint, you know? So they go out, get VLM, try running models on it, and it runs pretty fine. But as you start serving these workloads, as you start scaling these workloads to bigger customers, You'll realize that the base, just like basic VLLM as you like, is just not good enough. That's why companies like Radix Arc, companies like the folks who created VLLM, they also have a company, I forgot the name, it's on—

Infraract.

Yeah, Infraract, all those folks. That's why they're creating companies in order to make kernels as well as develop megakernels, which reduces this launch kernel time. In fact, giving Faster inference, like inference latencies, you know? So of course you cannot just go out, install VLM. Again, that's also like a huge pain to do for a lot of people, but you just cannot go out, install VLM, run it on a cluster and like assume to make 80% margins, you know, as well as like making sure like your cache-aware routing is in place, et cetera, et cetera. So there's a lot of things. There are a lot of moving parts here.

Okay. Can you, can you tell me about coming up to speed on using clusters and the relationship that you have with agentic coding at this point? Because what we've heard from providers is that agentic coding has been like very difficult for them to contend with. It doesn't, it's not a free lunch where you, everything just works out of the box and the model can solve everything, though it can be quite helpful for debugging. I guess the high-level lead-in for you is that agentic coding is like resulting in a different type of stress on the system. 'Cause you can just like have these coding models get away from you in terms of what they're launching on these clusters. You have some of that experience that you can share?

For sure. It's very interesting because a lot of our ClusterMax testing, we lead with our CLI, C-Max CLI. Please check it out. It's open source. Feel free to try it out on your managed cluster, play around with it. Check if you want to contribute, feel free to contribute. Make sure to read it before contributing. But of course, of course, we do test our— we do use agentic coding for testing all these clusters. And the workload is very different. The workload is very stressful on these systems, you know? So that's why, like, we have cMax CLI as a front, you can say, like, a front end, and, like, the agent is just, like, a wrapper around it. So if there are any issues which cMax CLI faces, the agent in turn, like, debugs into it and understands what's going on. But yeah, that's like the main use of agentic coding, even in like day-to-day, like my work. We'll love to hear what Sam does. He also is a huge user of Astra, right? Is that right? Are you Astra or Fable?

I'm a 5.6 SOL guy. I find, I find Azure just so badly undercooked. I get more done with Five6 right now, but certainly not Claude. The word on the street is that they fixed the writing, but I'll believe it when I see it. Yeah, I mean, anyone who's used these tools extensively has an idea of how frustrating they can be. And I think that cluster orchestration is like the perfect sweet spot of like, close enough for the agents to be superficially useful and far enough that they'll just blow your foot off every single time. So it's not quite like— yeah, Jordan's laughing the laugh of a man who's had late night sessions where Claude is reliably leading him astray.

And I'm not saying— And use some vibe-coded products, vibe-coded infrastructure products, which is, you know, I shouldn't be laughing at it too much because it's not like I've been able to produce any reliable dashboards so far. Mine are plenty brittle themselves. But yeah, it's just funny. It's funny to me to think about getting tagged by Pratt in Slack of like, what do you think of this run? Well, let's check with computer what we think it'll cost on the endpoints. And it comes back and goes, yeah, this'll range from $300,000 to $400,000.

That was insane.

We do a full sweep across everyone.

I'm like, hey, that's light work. That's light work for semi-analysis.

Let's hold off on this one for now. Let's just make sure we're bulletproof before we start launching some of these runs. So anyway, yeah, like it gives you sort of a false sense of confidence and you need to maintain the discipline to actually try to understand how these systems work. And I think what we've seen is that the providers themselves have developed some products That they seem to not understand as deeply as they did a year ago, despite the fact that the product is not meaningfully more complicated than the one that they were providing a year ago. It's like, there's no, we're reaching this wall with clusters where the top providers, there's not that many more features that they can add. They need to contend with bringing online the latest and greatest GPUs fastest, which is, you know, complicated in and of itself. But the expansion opportunity is, you know, bare metal data centers becoming a construction company or like an operations company, basically. Inference endpoints, RL training infrastructure, which we can talk about a little bit, but spin up a few VMs and create a Kubernetes cluster should be a solved problem despite the fact that, well, the agents haven't quite solved it yet. Yeah.

Yeah, I think that's right. I mean, There's a handful of somewhat high margin offerings that we can think of, you know, some good providers not yet offering, I guess, performance storage or block storage are the 2 main ones that are not offered by everyone that seem like, you know, an easy way for them to expand in sort of their current shape. But I completely agree that there's plenty of Plenty of info that, yeah, feels like it should be right in the sweet spot of the models. There's, you know, it's like plausibly properly configured, but if you spend as much time on clusters as we have, we know exactly how we want it, or at least we know exactly how we don't want it. And we see plenty of setups that we don't like. And I think that honestly, putting the cluster max write-up that we have here into your agent's context and telling it to rearrange your orchestration is better than anything else I could think to mention. Because, you know, there's like a lot of good patterns in the training data. There's a lot of ones that just don't work for the use case that we're speaking about right now. Same for Kubernetes, same for, you know, Knickel configuration. And I think that, yeah, there's a lot of people who have sort of outkicked their coverage trying to set up stuff that they don't actually have experience with. They haven't spoken to customers. You know, it's not easy to Google what the right way to set up this part of a Solenki cluster is, and your agent will do it for you better than you could have done on your own, but it will do it incorrectly.

Yeah. And let me, let me give like a real example of where this kind of manifests with a little quick demo of our, our dashboard and just some troubleshooting. So not going to pick too hard on these guys, but Firmus was a company that we worked, you know, pretty hard with on the GB300 to actually get it working. And what I've got on screen is one of the example dashboards that we track, which is a nickel performance curve that Sam was describing earlier. On here, I'm highlighting Firmus, which is like, there's 2 runs, one that we did after a bunch of work together and one that was done earlier. And so the Chart on screen shows time, which is like the latency for a message to actually pass over the network in this all-to-all collective. And more time is bad, so lower the better. And this is clearly an issue with the network configuration that we identified. What happens is that if we get a collective constrained workload, which we do have here, and I can go down to our TorchTitan GPT-2 SS training job, which is run with expert parallelism and is communications bound, is that you see the performance on Firmus out of the box was 1,455 tokens per second per GPU, whereas the others on some testing were up around 3,500. So way, way less performance out of the box, like 50%, which is an issue. And if you drill down into these steps with some of our profiling, this yellow bar here is exposed communication time that you're not supposed to be seeing. And so just like backing out and showing all of these providers highlighted together, like Firmus, obviously on a per-step basis is just taking way longer. And that's because all of these exposed comms are actually being shown in there. So anyway, this is, this is not a bad provider. This is a company that has stood up thousands and thousands of GB300s. They have happy customers. They were eventually able to fix this with us after we did some testing, but it's an example of like, A company that has not built the battle scars of working through that back and forth with a customer is not going to have an idea of how to set up a network correctly for the GPUs out of the box. And we just default to trusting people who have experience with the GPUs that you care about the most, and that's the latest and greatest GB200, GB300 system. Who are you going to trust for your Vera Rubin going into next year? Probably the people who have been through those battle scars before. And know what they can trust from NVIDIA, what they can trust from their other vendors in the documentation, and what they maybe they can't. Okay, let's, let's move on from networking and talk a little bit about storage. Sam, you had a fun experience with one of the providers trying to get them to get their storage to work correctly, which specifically led to a conversation about security. Maybe you can tell us a little, little story about that experience.

Sure. This is a story with multiple layers, I guess. We got an email notification. This is the one that you're thinking of, Jordan. We got an email notification for a node down. So as mentioned earlier, we do synthetic injections of XIDs to test the health checks, but we normally do this at the end of testing just in case it doesn't come back. We don't want to run shorthanded for all of testing. So we got an email notification that a node was down pretty early on. So screenshot it, put it in the Slack. Is this anyone? Nope, it's not me. So—

Jordan, are you messing with my cluster? No, man, that's real.

Yeah. Jordan, what are you doing, dude? We're only gonna have 3 nodes here for the rest of testing. Turned out that it was an actual failure, and it was a pretty fun failure. There's some way that, like, working on ClusterMax is the exact opposite of being a performance engineer at a lab. Because if you're a performance engineer at a lab, you just want everything to work. And here, like, we want bugs, otherwise we have nothing to do. So this was like one of the better bugs that we saw in training. Yeah, I think if I like met an actual perf engineer and shook hands with him, we would both disappear. Engineer with anti-engineer. So got on the cluster. And there's a lot of logs to dig through. This is, you know, an example of agentic coding being helpful. We weren't sure exactly what happened. No one was actually running anything on the cluster. It turned out, long story slightly shorter, that there was an error with the Linux kernel. So the Linux kernel had a hard time dealing with reassigning dirty inodes. So you You know, you have a bunch of stuff written in the writeback cache, and you need, and they change, it changes ownership. So in our case, this was like an old Slurm job that was ending, so we needed to deprecate it. This is like the behind-the-scenes stuff of an operating system that no one ever cares about or wants to care about. This was a very old Linux version, you know, about, I think the kernel that we were dealing with was a couple years old and had bugs that had been solved fairly long ago. But there's like a bad, I won't go into all the details of it, but a bad quadratic time algorithm that was dealing with resigning all these bad, these 30 pages of memory. This locked up the CPU, all the cores corresponding to that NUMA node. I'm not sure exactly why it was constrained to one NUMA node, but it locked up all the CPUs corresponding to that NUMA node as it was tearing down this LIRM job. The client for storage couldn't get any CPU time. So there's an ordinary request like, hey, are you still alive? To the Lustre clients, it didn't respond. So that triggered the health check. It said, you know, storage is failing, but the health check couldn't actually complete its work because the CPU was frozen up. So it took a couple of hours for that job to complete. Finally, the CPU actually returned to healthy because it was done doing its laundry. And as soon as it returned to healthy, it was like the cops were waiting outside its door. And it got health checked. So that was the cause of the email. And yeah, I mean, the last thing that you want is a node that's getting taken from your fleet, not because the GPU is actually unhealthy, but because of a health check failure for your file system that was due to a failure with your CPU. So that was a fairly obscure corner case with a file system. It meant that, you know, I was burning all kinds of Codex tokens over the weekend, setting goals to try to figure out exactly the right workload to recreate this bug. But in any case, yeah, we want the storage client to get CPU time is one way of summarizing our conclusion from that event.

Yeah. And actually, this turned into a security issue too, right? Which I've talked about on a couple of episodes previously now about the asymmetric nature of like when you're trying to find a bug, If other people have more powerful models than you, you can't defend yourself. Prat, maybe you can tell us a little bit about digging into security on a couple of these clusters where you're trying to identify if you were protected against a given vulnerability. And in order to even start asking questions, a lot of the models would just completely refuse to help you, right?

Yeah, I mean, that's, that's one part of it. But the other part of it is that, and this is something I've noticed, is that Frontier models especially just don't want to, like, exploit vulnerabilities. They just have a tendency of, like, showing false positives, being like, yeah, everything's perfect. Like, the other day, I was testing out, like, a cluster and basically just playing, playing with it, making sure— there was a vulnerability that I wanted to exploit, and I was using, I believe, one of the Frontier models for that. And the model said, yeah, we have this bug, we can exploit it, and we can actually get host access. host access to the control plane of that cluster. So I was, like, overjoyed, you know? I was like, yo, this is a perfect, perfect chance for me to, like, show that the CVE is actually a really, real big thing. However, the model actually produced a false PoC, and after, like, 2 hours of debugging, the model said, oh yeah, actually, my bad. Like, this was something I accidentally, like, said. This cannot be exploited. Now, you could think, like, those 2 hours were wasted, But I would say I learned a lot at the same time. And this is something that the frontier models especially do. They just want you to believe in false positives. And that makes me realize that, are hallucinations gone? Most likely, yes. But are false positives gone? Definitely not. So that's my fun experience testing these clusters and just exploiting these CVEs.

Yeah, man. Lots more work to be done there. Sam, what's coming to mind when it comes to security? We've talked about this on a previous episode, but a few things have happened since then, including OpenAI being hacked. Part of a bug bounty program, the HackTron AI guys opened a PR in their internal repo. Seems like they're vibe coding because their monorepo had 1.1 million PRs or something. So they put the million and one PR up to prove that they had full access. Um, a little crazy, but they are, uh, one of many that are actively pursuing the, uh, these bug bounty programs, which not a lot of these NeoClouds are participating in right now.

Yeah, I, uh, certainly have a rant about this loaded. Maybe I'll, uh, I'll keep it in my pocket for the time being. I think that, you know, whatever your opinions about AI doom or about, you know, whether we should keep pushing full speed ahead, the fact of the matter is that these are extremely valuable, extremely powerful pieces of equipment that are very badly secured in many cases, very carelessly secured. You know, there's nothing like a standard set of tests that you can run to make sure that your cluster is properly configured. You know, things are moving very quickly. I understand that there's, you know, a million things on the to-do list of the people working at these companies. At the same time, we see, you know, CVEs that are 2 years old that apply to, like, the basic software that's running on these things. You know, we're not asking for super fancy things when we want tenant isolation. You know, we want not to be able to see what's going on in the Grafana of our neighbors. And we don't want, you know, CUDA Toolkit 12.0 when 13.2 or whatever it is, is the most recent one. So, I mean, yeah, apocalyptic scenarios aside, they're like just very straightforward ways that you can see this being a tinderbox for these tools that are only getting more powerful.

Yeah, we talk about agentic AI being needed to be used to defend against agentic AI attackers, which certainly seems real when you go through the details of the Hugging Face incident. And also feels real when I'm trying to understand the details of that incident or anything else that we're working on on security side, where it's just like the speed at which we have to move means you have to use these models to understand what's actually going on and comb through logs and all of that stuff. But there's also some stuff that you can do that's pretty straightforward, right?

Yeah, I agree with that. I was going to push back a touch. I mean, I understand that in the case of the Hugging Face incident, I need to stop using the euphemism supplied by OpenAI's lawyers, Hugging Face incident. It was the Hugging Face cyberattack by OpenAI. And it's strange to me that we've forgotten this in favor of this anodyne phrase. The Hugging Face cyberattack consisted of dozens of very mundane misconfigurations by OpenAI, almost inexplicably mundane misconfigurations by OpenAI. Sure, you have petabytes or whatever it is of chain of thought to dig through if you're interested in understanding qualitatively this novel model behavior. But you don't need an LLM to understand that you shouldn't have an applicable CVE to your Linux kernel, which is what the— was the first thing that the agents used to escalate their privilege. You shouldn't over— you should be careful about managing permissions on the Kubernetes layer, which is another thing that the agents exploited. If you want to sandbox your agents, you should make sure that you have Alerting if they have unmitigated access to the internet, which OpenAI did not have. So I think certainly agents can be helpful. There are ways that you can only fight slop with slop, but there's also just a lot of basic things that OpenAI didn't do and that these providers are not doing. And I don't think that we should allow them to have the excuse of, it's a brave new world. We're not the most skilled people with Quen 3, so you can't blame us for having bad infra. It's a lot more basic than that. And that's, you know, the fairly low bar that we're trying to hold everyone to.

Nice take, Pratt. You ready to fight slop with slop?

I'm down. Slop against slop. Let's do it.

Right. What's coming to mind, guys? What have we not covered? On this one so far? Any stories that you think we should be talking about?

Can we, can we cover the financing? Like, I believe it would be good for the audience to understand why is ClusterMax even important? Are we just going, testing out clusters, giving them feedback and calling it a day? Like, why is ClusterMax so necessary for this like $7 trillion buildout, which is going to happen or happening, I guess?

Good question. I'll take that one. Yeah. I mean, financing is like the biggest topic from everybody because everybody, like whatever you were doing 3 years ago, everybody is being asked to do it 100 times bigger right now. Everybody's being asked that question of like, can you take what you did at 10 megawatts and try to do it at a gigawatt scale? And what's going to break if we try and go that size. The thing that is breaking right now is that people, there is more money than good investment opportunities, except the money is still concerned about quality. And there's 2 big reasons why somebody might be concerned about the financing of a NeoCloud. One is if They're late delivering on their contract, so they don't build everything fast enough. They can't get it online fast enough and they can't hand it over and have the customer start paying, so they can't pass acceptance. And then the second one is if there's so much of violations of the SLA, so much downtime that the customer has a cancellation right and wants to cancel their contract. And so the simple dynamic that's happening right now is that people are trying to insure against these things. They're trying to insure against delays and they're trying to insure against contract cancellation. This is both the debt providers as well as the NeoClouds themselves that want to have insurance against these things happening in their customer contracts. And there are products for this. Companies like Parametrics and others that we're aware of are involved in providing insurance. They're underwriting the parametric risk of these contracts. In order to do that, they need actuarial input. So they need to have some sort of information about the cluster that people have, things like redundant power at the site level, redundant cooling, redundant internet connections, that they have the ability to mitigate failures, that they have designed things up to the performance standard that meets these performances, that they have a parts depot nearby so that if there's a bunch of failures, they can actually get things back online. That they have the correct amount of people on site, that the site is not in a floodplain or about to get taken out by a hurricane or something. And if it is, that they have the requisite insurance against those natural disasters and there's force majeure clauses in the contract that protect them. And anyway, on and on. And so the point is that we have developed a number of example SLAs and example contracts that we are circulating with customers of NeoClouds and both the clouds and the labs want this industry standard. Like they want some standardization so that they can make some customizations, but they know what the definitions of important terms are within these contracts. What is the definition of node versus rack versus site versus cluster? What is downtime? What is force majeure, et cetera, et cetera? There's all sorts of specific things and Because we've seen so many of these deals happen and so many of these contracts, we're able to, I think, provide a level of certainty, both the lenders and to the providers and to the buyers that they can get these deals done and that people aren't going to be having a bunch of issues later. With that said, the motivation for this is concerning. The fact that we've seen a bunch of people sue each other. We don't want this to happen, guys. Like at a very basic level, when companies raise hundreds of millions of dollars sometimes and they spend, I know companies who have spent literally 100% of their seed round on compute because it's a couple of guys that have equity in the business and they're not taking a salary yet. So they just get the check from the VCs and they turn it around and spend it directly on new cloud capacity. This is the most critical decision you can make in the lifecycle of your company. So it really depends who you choose to trust. So, you know, the offering from SemiAnalysis is really to provide a technical assessment of all of these providers, the decisions that they've made, their ability to respond to things. This includes hands-on experience. It includes visiting sites. It includes like doing the dirty work to get those details fleshed out. And yeah, clustermax@semianalysis.com if you want to Engage us for that technical assessment so you can underwrite your deal or you can move forward with confidence. But that's sort of the backdrop on financing. Maybe the other big conversation is that SemiAnalysis is launching a credit and markets model covering really the entire backstop universe that's going on right now. We had a great article that Dan put out Covering these details, Nvidia's backstop universe off balance sheet has grown to be over $588 billion that they're backstopping in the current, like at the end of FY27, basically. So the current or soon to be current fiscal year. The forecasts, if this continues, is for this off-balance sheet backstops to grow to over $2 trillion by the end of 2031. And this includes a whole bunch of detail that, you know, we published when it comes to actually building a site and who is being backstopped. What do they need to actually build? What is the likelihood that it all comes through? What's the likelihood that you have some issues actually raising the capital or Delivering on the commitment to the paying customer and things like that. But maybe the point is like, if you are interested in the financing of this buildout, go read that article, NVIDIA Backstop Universe, and contact us about the credit markets model because the amount of capital that's flowing into this ecosystem is worth paying attention to. Let's leave it at that.

Yeah, Nvidia, Nvidia, I believe, and I might get these numbers wrong, but are backstopping approximately $827 billion projected according to our model, $827 billion in fiscal year '31 only for the supply chain for like HBM, etc. This does not include like LPS. This does not include actually like the, like the stuff that like the data center buildout, etc. This is purely just the supply chain that they're backstopping, like HBM providers, et cetera, which is quite insane. It's $827 billion. Yeah.

Yeah.

There's a lot of stuff happening in the physical world in order for this stuff to be turned on, especially on time. But I think it's not just Nvidia. Maybe another point to make here is just that Amazon's doing this, Google is doing this. A lot of companies are putting their balance sheet to work because they want to be exposed to the upside without actually taking up a whole bunch of the, not necessarily risk because they're taking on some risk, but the quantity of risk that others are taking. And so a lot of companies are benefiting from this because they don't necessarily need to have a business built on Having investment-grade off-takers like Nvidia, they just need to have that backstop where Nvidia has confidence that these customers are going to show up and be ready to pay. And if they don't, you know, they'll take the capacity for themselves. But it's a pretty, it's a track record of companies showing up to pay right now. And companies that don't necessarily have the funding raised today who want to commit to a 1,000 GPU contract for a year don't have the money to do it for 3, 4, 5 years. But when they train that model in the first year, They're expecting to be able to raise more money and things are going to work. And a lot of these companies have to work for this to work out, but so far so good. We are seeing a lot of companies across all sorts of different industries, you know, robotics, drug discovery, material science, like they're producing incredible results, let alone the coding agents. So there's some, there's something important going on here. And I think it's, it's worth digging into and understanding the full extent to which Nvidia's backstopping things. Maybe the other thing to end on is just how much we are seeing old GPUs be used. Like lots of people want to give us H100s today, which are 3 or 4 years old at this point. Some moving into 5 years next year when we get Vera Rubin, we're going to have 5-year-old H100s on the market. This is not going the way that Dr. Burry said it was going to when he started criticizing everybody's depreciation schedules. There are 4-year contracts being signed today for H100 capacity. OpenAI has never given up an A100 GPU that they've gotten. Those came back in 2020. So those are 6+ years old. And maybe the question on everybody's mind and the finance side, Is terminal value. How can you underwrite the terminal value at 5 years or 6 years on some of these chips when these contracts expire? So far so good. If demand continues to outstrip supply, we will expect that these prices are going to continue to rise and people are going to continue to pay for even the old GPUs so long as they can be turned on. Tune in more when we, when we have another follow-up coming up soon about 800-volt DC and all of the modifications to data centers. If somebody, if people are going to turn on Grace Blackwell, GB300, they're running racks that are 140 kilowatts approaching 200 kilowatts at peak demand, right? This is way different than an H100 rack at 30 kilowatts where the entire facility is built for 30 to 40 kilowatts. It can't accept any of the new VeraRubins that are coming next year. And so these facilities, like what are they going to do other than host these GPUs? Maybe rip them out and put in new CPU servers for the sandboxes, maybe install some storage, but likely they just keep running the H100s that are so valuable until they expire. Anyway, guys, where do we go from here? Lots to, lots to talk about beyond financing. You want to talk about sandboxes or harnesses or RL stuff, Pratt? What are you going to be testing?

Should we talk about hosted training?

Yeah. Hosted training.

Like, Let's do it.

The thing I didn't realize until this week is that RL seemed like smaller because it's smaller than pre-training, but it's on a reliability perspective and like a correctness perspective, it's actually way harder because you're combining all the hard parts of inference with all the hard parts of training, and then you need to sync them up. Plus you add this thing called an environment or a sandbox that you also need to figure out how to deploy. So there's like 3 components that sync up as opposed to like just one thing you can reason about. Even if the post-training clusters are not necessarily so big. So what do you think about hosted training? Is it going to be a real thing that when people— we're not necessarily seeing everybody nail Kubernetes and Slurm, so now they got to do Kubernetes and then do a whole bunch of stuff on top of it. What do you think?

I mean, hosted training is very interesting. We've been talking to a lot of people. I still don't think there's a market for it yet. I said yet, so please don't come back to me if there is a market. But of course, one common thing you can think is that your IP, your intellectual property that you're training these models on, has to remain to yourself. It has to remain for yourself rather than going to a third party like OpenAI or Anthropic. This could be a completely different case when OpenAI or Anthropic just deliver a better product and you're just you're just like, I need a better product over protecting my IP. So I think that's going to be a good discussion to have in the near future. Another thing I want to discuss about hosted training is how infrastructure intensive it is. We have been— I was talking to this other evaluation slash this RL environments lab the other day, and they mentioned they literally have server farms, just running environments, you know, and just a whole room full of server farms, just running environments. And you would assume that, why don't we just run Docker? Which of course they do. They do virtualization to a decent extent, but at the same time, you also need to run it on like different hardware, like iPhones, Android phones, et cetera. So, which is quite interesting, you know, like it's so infrastructure, like intensive. That is it that once you take hosted training, you're just managing more infrastructure than you did when you were like doing managed clusters. Because now you have to autoscale the environments, you have to autoscale the inference nodes, you have to autoscale the trainer nodes. And at the same time, doing all of this, you have to make sure that there is no trainer-inference mismatch. So it's a million things to take care of. We should give more credit to these hosted training providers that There are like a billion things they have to manage. And my hot take is that managed hosted training is only going to become big if your company, if your startup is handling literally everything. And once you handle every single part from environment creation, literally sitting with your customer, deploying forward engineers, building environments, training, helping your customers train their models, and once a customer develops this, you know, like develops this, basically gets educated on how to like really work with this, then only you can take your hands back and like that entire run like happens. So this is a hot take I have, you know, you have to basically just carry their hand throughout the process.

Yeah. So, so dig into that a little bit more in the pre-training world. There's like basically no examples where the provider that's selling the cluster actually really understands pre-training. It's not like you contract a cluster with CoreWeave and they're giving you tips on data mixes and architecture and like, hey, I heard this stuff about sparsity. They're going to work on the infrastructure layer, which means performance, which means reliability. So what's the analogy on RL? Not only is the infrastructure harder, you also need to have forward-deployed engineers and understand this company's business and guarantee that the model you're training is going to work. That was never a commitment that Cohere had to make or Nebula had to make to one of their customers that like the training job was going to work well.

Yeah, I mean, it's an interesting thing, you know, like they didn't— people assume that posterior training is going to be very easy. Honestly, I assumed that too. I was like, you just train the model, the generator generates the model, generates the trajectories and just train the trainer on these trajectories. Voilà, you're done, right? But turns out the real valuable tasks in day-to-day work, which we do, is way more complicated than just generating trajectories or building environments. I think building environments is a very hard task, and especially that's why Anthropic bought the most environments last year. That's why their coding model was so damn good, is because it's a hard thing, and they paid a lot of money for that. So that's why they didn't expect it. A lot of hosted training providers basically thought that it's like managed endpoints or managed inference, that you just provide an endpoint, which still, of course, is not like an amazing experience. But they basically thought that you just provide the infrastructure and the client enjoys it. But there's a lot of education to be done. Of course, Rome wasn't built in a day. We still have more time. We're just in the beginning of this revolution. So yeah, we'll wait and watch what happens.

Last question about post-training. In the inference world, if you're making an analogy to that, there were some open-source frameworks that just dominated. And so, VLLM and SGLang kind of simplified things for everybody. We got some really high-quality experiences really quickly, and then people built a whole bunch of more complicated stuff over time with Prefilled decode disaggregation, training speculators, quantization stuff, cache-aware routing, all this extra bells and whistles beyond what the basic recipe from the, beyond the kernels, beyond great performance out of the open source models. Let's compare that to frameworks in RL. Slime, Miles, VRL, PrimeRL. What are you Seeing people use it, there's a lot of them out there right now. And some of the hosted training providers kind of abstract that away too, where you don't even know what framework you're using. You're just bringing your dataset, or even you don't even know about that because the forward deployed researcher, FDR, not FDE.

FDR.

We can't be telling these RL guys they're engineers and FDEs, right? They are FDRs. So, Yeah. Tell me about framework wars and what you think is going to actually show up there.

Yeah. Framework wars are very interesting because it's a very funny thing that all these inference providers, they started with providing amazing frameworks like SGLang, VLM, et cetera. But the next market that they thought to expand was basically just hosted training. That's how you can see Radix Arc capitalizing on Miles, which is absolutely amazing, by the way. They have made it very agent-friendly that you can just set it up in one shot. It's pretty decent. Of course, you have to set the hyperparameters, so it's not that easy of a task to just train a model, but they have done a very good job. Prime RL is another very good one. When I was GPU poor before joining SemiAnalysis— shout out, join SemiAnalysis— I did run a lot of runs on Prime RL, and that was also a pretty pleasant experience. Over time, my experience with these frameworks have gotten better. Of course, these teams have put in a lot of investments, and building these training frameworks is way harder than people think. I suggest the audience to read the Human Sandbox blog, where they basically show how native NVFP4 training is done at such a large scale, because trainer-inference mismatch compounds. Over steps. And as your trajectories get longer, you do want the optimizer to learn more technically, right? So you have to take a bit more steps. And if your trainer-inference mismatch compounds over like 100 steps, you'll realize that your model is diverging a lot and that collapses training. So shout out, shout out, huge shout out to all these framework providers putting in work day and night. Sending in PRs, even though it might be just pure slop. But you shout out to them.

Slop PRs, shout out. Shout out slop PRs. Yeah, yeah, yeah.

We got a term. We got a term slop PRs, like slars or something, you know, like like Neo Clouds, you know.

So yeah, I was at ML Sys earlier this year, and the one of the talks Roger from Infract shout out Roger was basically like. If you want us to merge your PR, you got to message us in Slack because we're not reading every single one of your slot PRs anymore. I was like, all right, good advice. Good to know. Let's move to wrap here, guys. Sam, what's, what's on, on your mind as we finish up talking about NeoClouds? Where do we go from here, buddy? What's next?

Sam, you want to talk about Azure?

No, no, we need to run it back.

Let's go company by company and criticize their internal organization, beginning with Azure and AWS. I want fewer obscure bugs. I want nothing to write about in ClusterMax 4.0. The better the cluster, the less there is to say. So I want to log on to Slinky and just have it all work, have topology awareness, not blow my foot off every single time I try to see whether the health checks work. And yeah, I hope that all my performance engineer friends get good clusters that they never have to think about.

No more providers randomly wiping your nodes, randomly rebooting them, randomly deciding that kubectl is going to go down for 3 days. We've heard, we've heard some horror stories from guys, right? So, um, yeah.

Protect yourself with a good SLA, but better than that, just go with a good provider. You'd rather not be fighting to get back your, uh, your whatever, 5%, because it was down for 3 weeks out of the year.

Yeah. Trust people. Let's give everybody a way in which you can trust everybody else. Join hands and sing Kumbaya. Thanks everybody for joining this podcast, listening to us talk about ClusterMax, and we'll see you at ClusterMax 4.0 after we get some VeraRubins in these data centers. Looking forward to it, guys.

What about, what about TPUs? Are we ever doing it?

TPUs on ClusterMax? Ooh, you're giving away too many secrets, Brad. Too many secrets. TPUs, Trainium. Some other novel accelerators that are being deployed in data centers across the country right now and around the world. Check back in, everybody. All right, Akash, cut that whole shit.