Dylan Patel (SemiAnalysis): The Datacenter in 2026: CPUs, RL Environments & Agent-Driven Workloads

Hello everyone. I'm happy to have Dylan here. Uh, the first time I saw one of his videos was an interview when he talked about, although we get to CPUs, we talk about like NeoClouds and why they have the right to exist. And that was super interesting for me because the theme of today is all like, what are the new infrastructure primitives? now that agents are here. And so you articulated that really well at that point, why neoclouds are different from hyperscalers and why they should exist. So can you share any of that with us?

Sure. I mean, um, in the age of AI, the hyperscalers were a bit slow to move, right? Google, Amazon, Microsoft, bit slow to move into AI. And so a whole new crop of companies, uh, popped up and there was a new low bar, right? There's no need for a lot of the complex software that Amazon, Microsoft, Google had built up. And a lot of this in fact slowed down AI, right? They had custom networks that were actually not that great for AI. They were more focused around reliability and storage traffic rather than, you know, doing an all-reduce across a network or something like this. So, so there was a lot of things that these large cloud companies, these hyperscalers had that NeoClouds could just skip, right? And, and then build focused optimized solutions and deliver much lower cost because they have much lower overhead, right? There's not 20,000 Google PMs Yeah. Sitting in a meeting, uh, at these NeoClouds, right? Um, although some of them have started hiring Google PMs and slowed down. Um, they, the, the NeoClouds, um, they, they, they just moved fast on energy. They moved fast on bringing up GPU clusters. And so they were able to carve out a market, right? Um, and then, you know, those are the early ones. And then since then there have been a lot more copycats or followers, um, many who haven't succeeded, many who are succeeding, right? Um, it really is like a, a battle of like who's the most competent.

And so, um, are all the— 'cause there's like, I think you said there's like an order, there's like 200 of these NeoClouds or, or give or take, right? And so are there differentiations between them that you see? Like, are, are some of them just copying the, the sort of software stack that these first ones did? Are they doing something else? Like anything that you've seen that have been successful or less successful across these NeoClouds?

Yeah, so there's a variety of things that separate them, right? Um, so we have something called ClusterMax that ranks all the Neo clouds, but we test for various things such as observability, reliability, networking security, orchestration. All these sorts of things are different, right? Like, does someone test for their GPUs to be working properly when the GPUs are idle for a user, right?

Mm-hmm.

That's an active health check or a passive health check. Hey, are the fans spinning at the right speed? Are the— is it drawing the right power? Is there something wrong with the node? Is the network flapping? Or is anything— problem with the network? Is the performance there, right? Like there's all these sorts of checks and tests because GPUs are unreliable. There's all sorts of stuff around the types of software you run on top of the GPU, right? Many people started off doing just bare metal, right?

Yeah.

So when you go look at like, hey, Microsoft's first contracts with CoreWeave, they were all bare metal, right? You just SSH in and Microsoft would set up their own stuff. But as we've stepped forward, there's a lot more people want, right? People wanted Slurm installed. Okay, that's simple. People wanted Kubernetes installed. That's a little bit more difficult, but still pretty simple. People wanted Slurm Kubernetes installed, um, because it's easier to push out jobs and things like that, right? Um, and now you're starting to see people do things like managed ray services and all these things for RL. Um, and so there is a class of neo-clouds that are building these things, and there's a class of neo-clouds that are like, I don't care about that, I'm just going to build GPUs and rent them out at bare metal. Um, and, and there is a difference in cost, right? Like the, the clouds that have the good software also tend to charge more Um, and, and sort of returns back to the old school model where Google, Microsoft, Amazon had good software and they charged more.

Charged more. Yeah.

Right. Um, and so, you know, and, and you see a lot of these cloud companies start trying to launch inference services, um, and all these other things.

Yeah. So like that, similarly in that vein, I'll just get to the, the CPU ones where historically we have customers that ask for this with like OpenCLaw that happened. So, there's a bunch of people that are like, oh, I need my sandbox or CPU box running for a very, very long time. Can you give me like a $5 Hetzner type offering? And I'm like, well, no, because like that is a bare metal machine that has no— incurs very little cost. But when you have something that's a larger software offering, then there is more cost. So it's hard to compete with that. So I would assume also on the NeoClouds that like the ones that are bare metal, as you mentioned, have a lower cost of sale than the software ones. That moving just— I was just curious about the, like, why the right to exist they had, because it's sort of analogized with what we do. But the real question is, you brought this up, which is directionally what we're thinking about, is that CPUs are the new bottleneck. Like, every investor partner, everyone that I talked to historically was just like GPU, GPU, GPU. And now you had this big report about, like, CPUs, and I'm like, okay, yes.

Yeah.

Thank you. And so you reported like this year will be on bottleneck. So just give us a high-level TDR. Why is CPU now bottleneck and what do you see?

Yeah, I mean, for the first few years of AI, CPU was a real big lagger, right? Sure, you used it for some storage, you used it for some checkpointing, used it for preprocessing your data and pre-training, but it was pretty light. And then inference was also like the models just weren't good enough to be agentic. You couldn't, you know, go step by step by step. So there was no capability for a model to go and do actions and string them together. So then there, it wasn't really, it was like, it was like you would send a string and it'd send a string back, right? Simple inference. And so there was not much need for CPUs, but then over the last few years or last couple years, it's not even like last couple years, right? You know, Q* OpenAI had that and they had all that drama, but then finally O1 preview launched, I wanna say like 15, 16 months ago. Feels like much longer ago.

Feels like a long time ago.

Yeah.

Ages. Yeah.

But 001 was the first model sort of there. And we sort of had an entire flood of models, right? Like before, people would do simple things like check the output of a model with regex or something simple just to see if it was doing struct— properly doing structured outputs for function calling and things like this, right? But as we've stepped forward, the checks on the model are much, much larger. And they've been integrated fully into training with reinforcement learning, right? Instead of just doing regex, you're doing various classifiers. Instead of doing classifiers, you're doing code unit tests and compilation. Instead of doing that, you're running agentic flows where it's actually calling databases and stuff, or you're interfacing with some environment that is heavy on CPU, like a physics simulation or biology simulation. You just kept stepping into more and more complicated things where the model outputs to CheXit, this environment, reinforcement learning environment, and then goes back and trains on it. loop has gotten tighter and tighter and tighter over the last couple years. And then more recently over the, let's call it really the last 6 months, um, you know, code agent revenue has gone from a couple billion to north of $10 billion in like a very short amount of time. Um, and, and these, the horizon of these has also increased dramatically, right? Like 5.4 codecs can work for like 6, 7 hours and it's like fine, right?

Yeah.

Um, and so You know, it's, it's in that timeframe, it's doing all sorts of calls to, um, databases. It's doing calls to, you know, at least we, we use a lot of cron servers, but, you know, like whatever, whatever it is, right? There's anything and everything it can go ping, check it out, scrape, and just work agentically on its own. Um, and that's also requiring a lot of CPU. So over the last 6 months, that's ballooned as well, in addition to the reinforcement learning training loop getting tighter and tighter. And so over the last 6 months, we've seen the entire cloud market run out of CPUs, right? I don't know if you folks have interacted with GitHub a lot recently, but it's really unstable.

I think you're the 3rd person to mention that today.

Okay. Yeah.

Yeah.

Yeah. So we've, we've been, we've been like checking GitHub's like stats on like how often is it down? How often does it fail to commit? Like, you know, whatever, right? It's, it's terrible. And that's because Microsoft sold all their CPUs that they had spare to other people, right? Either internal use for their lab, but not really, more so like external labs. They've signed deals with Anthropic and OpenAI. And so they just have no CPUs left, right? And we've seen the same at many other firms, right? So whereas before you'd have many GPU servers per CPU server, and so you'd be like 100 megawatts of GPUs would be served by even 1 megawatt or less of CPUs. Nowadays the ratio is like getting much, much closer, um, both for RL training and, and for inference, uh, agentic inference. So then you, you've just seen everyone run out of, um, CPUs. Amazon's volumes on CPUs, uh, of CPU servers they're installing have 3x year on year, right? From this year versus last year. Um, and so there's just no capacity anywhere and that's causing a lot of instability, um, not just in GitHub but probably other places too.

I mean, mo— a lot of infrastructure today, we talked about this, like every day it's normal to see that some infra provider, be it GitHub or being like an actual, not to name, like they go down. Like it's become quite a constant thing. And so could be lack of CPUs, but also scale of the workloads and whatnot.

So. Could be that everyone's infra code is vibe coded.

Yeah, it could be that everyone's infra code is vibe coded. Yeah, could that be as well? I don't think it's all, but there could be, there could be parts of that. that is there. And so what I see, which is interesting as well, is the number of CPU workloads or the number of that are running on us. So Daytona has basically 3 use cases. So it's like code and command execution. So if like a, a, a Claude code type of thing needs to run on a CPU, we have computer use use cases, which we see actually growing really fast. We've now launched today, we announced our Windows sandbox, which runs on a CPU. So if you need an agent to do like on legacy software, which is like all finance, all support, all, um, that's all there. But also to your point, like reinforcement learning, we have a bunch of people that usually use like Kubernetes. Now they would use us. But the interesting thing is that the, the size and scale of these loads are insanely large and they're growing insanely large. And we're like the smallest cloud in, in the world. So my interest to what, and the question is like, if we as a very, very small company have this much, what does that do at the large scale, right? Like, so And we've had, I'm curious if you have any insight on this, but just RL, let alone the long-running agents, which is on their own, we see customers coming in. One of them ran a million, one single customer yesterday in the span of 6 hours, like 1 million vCPU workloads. So that's one customer. And so like how many of these customers are doing RL that are gonna need this? I don't know if you have any insights on this, but I'm just curious.

I mean, there are some pretty shocking metrics, like a million vCPUs sounds insane, but some of the scale of the workloads and contracts people are signing are even like more ridiculous than that, right?

I'm sure that's because we're really small. That's why.

Yeah, yeah, yeah. So I guess like when you look at like, again, like when you look at like the Anthropics and OpenAIs of the world, they have fully just eaten the entire capacity of multiple clouds, right? A lot of the impetus for the recent Amazon and OpenAI deal was, yes, OpenAI wanted money. Yes, they needed compute, but they also just like went to Amazon and were like, give us your CPUs.

CPUs, yeah.

Right? And before OpenAI stack was pretty much only on x86 CPUs.

Yeah.

But Amazon had tons of Arm CPUs and so they ported the whole thing, right? It's like anywhere, I'll, I'll port my code base to get access, right? To access to CPUs. And so this is like the, the level of like, and it's not that difficult to port from x86 to Arm, but you know, like there's, there's very few dependencies that aren't working now. But this is the level of like engineering people are willing to go to, right? 'Cause usually devs are too lazy to do anything. And they'll just go get capacity elsewhere, but there just isn't capacity anywhere else.

Yeah. So, well, I mean, that's interesting. Like we're all x86, that's all we have. We don't even have ARM at the moment. But then like there's those 2, which I know, but also like Nvidia has their like CPU, other people building out their CPUs. There's also differences between these CPUs. Are there just gonna be like these general purpose CPUs? There's like, you probably know more about this than I do. I'm just like super curious.

On the types of CPUs?

Like now there's so many types. There used to be just like x86 and ARM. That was it. Like for the most part now there's different types of CPUs. Is it just because everyone's run out or are they like better at some things than others? Are there anything?

I mean, usually what happens like when there's a gold rush is the person with a broken pickaxe can also sell their pickaxes.

Yeah.

Um, but like, yeah, I mean, this, this CPU market is like quite dynamic, right? Um, you know, it was mostly Intel, AMD. I assume you mostly used Intel and AMD CPUs. Um, both of them have said they're like fully sold out. They've sent price increase notices to their customers. Um, so they're not even competing with each other anymore. They're just like, how many can I make and sell? Um, and likewise, like Amazon had their Graviton CPUs, they've had what, 5, 6 generations now. Um, and—

Nvidia has their Vera CPU, right?

And Nvidia has their Grace and Vera CPUs. Um, and no one really deployed Grace standalone CPU boxes. There was like a couple small deployments that Nvidia, Nvidia did for PR with people. But in reality there was like very few like standalone CPU deployments that Nvidia was able to do. Okay.

Um, why is that?

But now, well, just, they're just not as good, right?

Okay.

Um, but now as we step forward, maybe their CPU got better. Uh, maybe they're bundling better, but also just because they have capacity.

Because no one else has them.

Yeah. They, they're, they're able to get a lot more contracts on their, uh, Vera CPUs, uh, that are gonna deploy sort of later this year, early next year timeframe, um, is when they'd start deploying. So it's sort of like a very, um, dynamic market. And then, and then you've got like Microsoft and Google are starting to deploy their CPUs in volumes. Um, Arm is gonna launch a CPU in like a few weeks, uh, that Meta's gonna adopt and a few others are gonna adopt, um, like Cloudflare. And so there's gonna be a lot more proliferation, like Arm standalone rather than like Arm licensing the IP to someone. So there's a lot more, um, diversification in the market, um, which, which happens when there's a gold rush, right? And then we'll see what actually is the best quality and is left standing as the sort of supply-demand gap—

Gets back.

Closes. Yeah.

But it seems that the size of the band will still grow, like at least what I see, like one, the RL, and it, it seems like RL is mostly done, you know, they do post-training on RL, but we have now vendors and companies that are pitching and creating services that do RL in real time. And since you have like some agents, you have some SaaS that's an agent in the background, and then it will do RL at the end of the day basically to like learn against what it has done. And so that grows, but also these long-running agents, If they— the way it seems to be working directionally is if the agents can work longer and solve for more things, you can essentially ask them to do more, and they will spin up more and more of these agents, which means more and more CPU boxes. So do you see that, like, from your purview? Do you— like, I understand market dynamics, and that'll probably converge at some point, but I feel it's getting wider before it's shrinking, the need for that.

Yeah, I mean, that's totally the case because initially all the RL was like, let's do math proofs. And it's like, okay, math proofs are really low on resources. And also the model, right? The generator will generate tons of output and then it'll send the correct answer or what it thinks is the answer to the server. And then the server will verify it.

Fair enough.

But as we've moved forward, right? Like it's, it's not just that, it's like, oh, actually the model is committing many times or it's like trying to compile many times or it's trying to do unit tests. Throughout its agentic process. Okay, now that's increasing the frequency in which the generator, i.e., the model, is sending to the verifier, and that loop gets tighter and tighter and tighter. And as we continue to move to more complex RL, it's like the model will actually be verifying its output constantly, right? Like, if you imagine, let's say, models that are training over the next year or 2 is, let's say, a robot model that's verifying in the— that's in a world model. Right? So, uh, VLM is, is navigating across the world trying to pick up things.

Yeah.

Put 'em down. Well, every single action needs to be verified against the physics model that is sitting on some CPUs.

Yeah.

And so that's, that's insanely way more CPUs than you need for say doing unit tests or running a math proof, right? And when we look at like O-1, that's like all it could do was math, right? Um, and as we look at like, you know, sort of, uh, GPT-5.4 or Opus 4.6, like it's, OK, agentic software. But as we move forward to the next stage of whatever's coming, it's going to be models that can understand, oh, I need to tie my shoe. And when I tie it, what's the strength?

Strength, yeah.

Yeah, what's the tensile strength? All these things that has to be calculated because the verifier is just generating what's the next step. But every step needs to be checked way more often. And the intensity of that calculation for checking the step, it goes up a ton.

Yeah. But there's also another thing, and you probably know better than I, is like the level of the strength of the GPUs can handle in parallel a certain amount of CPU boxes. And so as we get next generations of GPUs, I feel that they'll be able to spin up, manage, or work on more CPUs than they do now, which adds another stressor on CPUs now.

Yeah, I mean, that's certainly the case, right? GPU, Well, GPUs are also just getting way higher power, right? So one GPU will equal many, many more CPUs—

CPUs than it does—

Over time, right? And GPUs are getting more expensive gen on gen on gen, whereas vCPUs are flat, slightly down.

Same.

Yeah. And so certainly, it's a different scale, right? One Blackwell versus one Rubin, performance is up x. Price is also up x. Whereas you may buy last generation CPU, new generation CPU, and there's 192 vCPUs in this gen and last gen was like 96, right? So now you've got way more CPUs, but the price increase is equivalent to what you got in additional vCPUs. So it's like sort of, yeah, the ratio will grow in the favor of vCPUs, but the cost will be sort of also probably in the direction of CPUs, but not sure how much. Yeah.

And also the other pressure on that, and we've seen this with larger customers when they have like, they don't, they have like a time allotment for GPUs and they don't want their GPUs to be idle. So they rather pay, uh, for warm pool running CPUs so that whenever tasks come from the GPU, the CPUs are red, they're hot, they're essentially working, right? So they'll, we, we, I say we 'cause we do mostly CPUs, are sort of like the cheap resource, although I don't think we're cheap, but cheap resource for this. And so that spends more GPUs actually just because the cost of having an idle GPU is to, it's way more expensive, right?

Yeah, that's actually like a really interesting point, right? The business models, like no one does, I mean sure there's on-demand GPUs, but like Lambda has like 50,000+ GPUs and only 4,000 of them are on demand, right? And they're always sold out. So like no one really has on-demand GPUs. Everyone's signing at minimum like long-term, you know, multi-month contracts, but in most cases multi-year contracts. And so like, CPUs have been used to like you can spin them up and down, right?

Yeah.

That's why everyone moved to the cloud, right? But with these RL workloads, it's like, okay, the GPU generate— the generator, the model running on a GPU generated a bunch of stuff. It sends it to the verifier. If the verifier's not ready and waiting—

Then it's idle, then it's waiting.

Right? Like the GPU's just sitting there. So it's like, okay, well I've already paid for this.

Yes.

So if you can't get your instantaneous, you know, resources spun up on the CPU side. Yeah. And in fact, you should do it preemptively so it's not like loading, you know, the, the simu— simulator that you've, or the generation, the environment.

Yeah.

Right. You, you, you're wasting money. So yeah, like that, that makes a lot of sense.

And not only that, then you have that running and then as soon as those are running, you're spinning up another warm pool of these so that every new iteration of them are, are continually growing. So what does that mean gen— generally? So we, we skipped over RAM, we didn't, memory, we didn't talk about that. where it's like GPU was a bottleneck, RAM, memory was a bottleneck, and now CPUs. So just very closer to home for most people is that like it's very hard to buy a PC today because it's like super expensive. Is, does that happen to the CPUs? Because we just mentioned that they're like, they're fairly inexpensive and flat. Does the market pressure now demand sort of that these prices now increase?

Yeah, I mean, it's, it's pretty much the case for all like PCs, laptops, building a PC that it's hard to get, you know, Apple was pretty sold out of like Mac Minis, for example.

Yeah.

We bought a bunch of Mac Minis 'cause people were switching from, you know, people who used a bunch of Excel and like Windows were like, okay, I wanna use cloud code. And OS X is obviously a better dev environment, right? So we just bought a bunch of Mac Minis and people were deploying them and using them. And sort of, I think similar has happened across the space. And then data centers are way more of an inelastic buyer. of resources. And so that's also caused prices to go up, right? So like GPUs always were expensive. Nvidia's margins have always been, you know, 70% plus. CPUs, their margins were not that high, but they're creeping up because Intel and AMD are raising prices. Um, and they're tight. Memory has like 4x in price over the last like year. Um, and it's gonna continue to go up. Um, and then now storage SSD, right? So it's like all of the resources are, uh, SSDs are also up like 3, 4x in price and, going to keep going up at least another 60%. Not as much as DRAM, but up a lot. And so between these things, it's like, well, the CPU capacity for Intel and AMD is somewhat fungible between— and substrate and all these things— somewhat fungible between PCs and data centers. RAM and storage are very fungible. And so you've ended up with this conundrum that, look, screw the bot— screw the normal—

Humans?

Screw the humans? Yeah. You have to buy a Mac Mini now. Otherwise, you'll never escape the permanent underclass is like the thought process.

And one last note, we're close to the end, but one thing, not financial advice, it's like Intel was in a very bad spot, like quite like recently.

Still are.

Yeah, but does this pull them out?

I mean, they'll do better, but like it's not like it's like a long-term, like, you know, oh, the company's saved, right? Company's valued on like future cash flows.

Yeah.

And also like some potentiality they get like Apple or someone else as a customer.

Yeah.

And this is actually like the funnier thing is it's not because CPU demand is so high. Intel will have some short-term profits from that, but, you know, others will catch up. AMD and Amazon and others will catch up on capacity that they have. More importantly is that AI is buying all the capacity on 3-nanometer, um, and 2-nanometer, uh, in a couple years that, you know, people are having to turn to other directions, right? Like one of the, like, You know, people make all these like nonsense reasons up why Nvidia acquired Groq. Um, part of it was because they wanna have really fast inference, but like part of it is that Groq is manufactured on Samsung, right?

Mm-hmm.

Um, because there's no 3-nanometer capacity for them at TSMC, they need to chip somewhere else. And if you just, if AI is as crazy as like we believe it is and demand is as crazy as we believe it is, it's gonna be even crazier next year. And so like just make any decent chip and it'll sell. Right? Is like sort of the philosophy there. Obviously there's a lot more they're doing there on the architecture and stuff. But same applies to like Apple is getting squeezed out, right? Like TSMC told them, hey, like get off of 3 nanometer, move to 2 nanometer faster.

Oh, not gonna do it. Yeah.

Right? Because all the AI chips are on 3 nanometer and it takes time, right? Small mobile chips are easier to make than big AI chips. So all the AI chips are moving to 3 nanometer now, right? MI350 from late last year from AMD or Trainium 3 and TPUv7 from late last year. or slash early this year from Amazon and Google, or Nvidia's launching Rubin next week, right? That's also—

We had GTC this week, yeah.

Yeah. So all these things are 3-nanometer, and they're just telling Apple to get off. They're telling Qualcomm and MediaTek to get off. And all 3 of these companies are like, oh, maybe we should use Intel because they're not telling us to get off because they can't manufacture edge chips. Yeah. So, it's pretty tough for everyone.

I have a bunch of questions I'd love to ask you. We're like, we have 20 seconds left, so the next question I ask you is probably going to go off. So, with that, I'll just like keep us on time and appreciate so much for coming and talking to us. Thank you so much. Awesome.

Thank you.