Ep. 034 - The Fight for Fast Tokens, TPU v7, Vera Rubin, and Engrams (AI Supply Chain, InferenceX)

Hello everyone. Welcome back to Semi-Analysis Weekly. I'm joined this week by the InferenceX team. It's been about 2 months since we've had them on the podcast to give a little bit of an update. Uh, this episode we're going to talk about engrams, um, offloading, AgentX, TPUs, and anything else that we think of. Guys have been hard at work, so I got Cam, Brian, and Alec. Guys, welcome to the show.

What's up? What's up?

What's up? Hello, nice to be here.

Awesome. Okay, so the latest article that came out, we're going to start with that one, came out September 18th, so almost 2 weeks ago at this point. It's all about n-grams and what happens when people co-design the model architecture for a capability of offloading certain parts of the model or the KVs to DRAM or SSD. So How about we just start by talking about what are n-grams and give a little bit of an introduction there?

Yeah.

Yeah.

I, so I think, you know, n-grams basically have been n-gram models and the concept of sort of like transferring more things to the embeddings have been around for a while. Brian can touch on more on like the history, but basically Uh, when you have a transformer architecture, each token has an embedding, right? And the embedding is some, you know, latent space of like meaning about what that token means. And as you know, you have a sequence of tokens and it, uh, you know, makes its way through the layers of the transformer. You can think of each layer as sort of building more information about like grounding the facts of what, of what that sequence means, right? As the tokens like relate to each other and so on and so forth. So like as it flows through the layers, more meaning flows into the hidden state. So the idea with n-gram is basically instead of having these layers, you can basically during training time, you can have the meaning of multiple. So at 2-grams and 3-grams. Token embeddings kind of learned at training time. So then these, these can be stored in a hash table and then looked up at inference time. And this is basically like for free. I mean, not for completely free, but if you go down to the next graph or a couple graphs down, you know, DeepSeek ran some experiments, right? And Basically on the right side at 100%, right? It's the loss for all Mo layers. So no n-gram layers. And then on the left side, it's all n-gram layers and no Mo layers, right? So you can see that the loss is actually minimized at like a middle point, right? So when you have some n-gram layers and then some Mo layers, and it turns out that the performance is actually quite good. So that's a high-level overview, but I think, Bryan, maybe you want to talk a little bit more about the motivations for n-gram and the history.

Yeah. So the paper that came out introducing DictSig earlier this year actually talked about the intuition for adding it, which was that models had to build up this representation of the meaning of words basically in later layers. So they tested this by comparing the next token prediction ability of the hidden state of the tokens at different layers. And there was an earlier paper on this, what's it called? And basically with n-grams, if you calculate the KL divergence between the hidden states of the early, later earlier layers and the later layers, you notice with n-grams, the earlier layers have lower KL divergence with the later layers, which kind of shows that the model is moving towards next token prediction earlier in the model instead of later, which is good because it means that you're not wasting parameters on memorizing stuff, memorizing or trying to build up the meaning of words. Yeah. And the co-design of this model, which is one thing that DeepSeek really specializes in, this knack for considering inferencing while designing the architecture of the model, is that n-gram models can be asynchronously overlapped with other parts of the model because the inputs for the n-gram modules are just the token IDs. So in their paper, the original paper, DeepSeek also talked about the trade-offs of n-grams being put earlier versus later. Because if you have n-grams put, n-gram layers put earlier in the model, they can very well, they can help the most. And the ablations show this, I believe. The n-gram layers help the most as it can bring in the memory of the meanings. Earlier on. But the problem with having the n-gram layers earlier on is that there's really nothing to overlap it, right? Because you have to straight away start waiting for the n-gram retriever to end before continuing. So this actually shows up in Quen's 3.8 Flash Next, I think. They put the n-grams in the second layer because the first layer gives it time for— retriever from DRAM or SSD to actually come in first so that there's no, there's very little overhead. But interesting part for V1, V4.1 Flash is that they put it at layer 1. So there was still space for overlap, but not as much. Yeah. And this brings me to the results, I guess, comparing, we have graphs in our article comparing the DRAM and SSD offload versions of Ngram. Jordan, could you pull up the chart?

Yeah, trying to find it.

Yeah, no worries. Yeah, but while Jordan is finding the charts, maybe I'll talk about like this offload, how the offload works. So the offload uses UVA, which stands for United— yeah, Unified Virtual Addressing, sorry. And basically it just moves the n-gram to CPU pinned host memory. And oh yes, thanks Jordan. And yeah, we can see that DRAM move. We actually tested HBM within HBM n-gram as well as, I don't think we talked about in the article because the results aren't very interesting, but HBM.

Take a step back here, Bryan. Like the implication of this is just that there's a significant HBM constraint and people are innovating around that constraint. by introducing n-grams to make it easier to offload to DRAM or to SSD, right? And this is just maybe a natural evolution of that progression over time. Is that fair to say?

Yeah. I guess one part of like—

I was muted. Bryan, maybe you have context on this. I don't know if necessarily n-gram was a specific co-design for restriction of HBM? Was it like, was that the approach?

Definitely partially. A lot of, yeah, a lot of DeepSeek stuff addressing like limited HBM, like last year even their whole MLOE thing was about reducing KV cache for HBM usage. And I think like this is along the same lines as well, because if you think about it from like first principles, you want something, you want a technique that can like be overlapped. So either retriever or— so yeah, so just retriever. And the stuff you get at the front of the model is basically the token IDs. So basically what can you do with token IDs? With only the token IDs, they can be added to the hidden states and overlap. And I think n-gram is a very natural consequence that you get to. Actually, the LongCAT team from Meituan, claimed to be concurrently working at the same, with the same technique. So it's actually quite cool that like 2 labs, I'm not sure if they collaborated or what, 2 labs arrived at this conclusion independently that n-grams was a good technique.

Yeah.

Anyway, going back to Jordan's point about HBM, I think we talked about this in our previous 4-High podcast, but Yeah, HBM is really a big constraint right now and we're trying to offload everything off of HBM, right? Like, uh, we heavy offloading. Firstly, I think Cam and Alec are quite, uh, experts in this field given that we did it for AgentX. Maybe, maybe they can explain about it.

Yeah, I mean, a big part of that, um, 12-high, 8-high, 4-high Rubin D spec conversation around HBM was the fact that the American labs are optimizing for HBM bandwidth over capacity or bandwidth per bit. In some ways, the Chinese labs have already had that constraint forced upon them by the chips that they have access to. And so, yeah, maybe you can talk a little bit about the Chinese chips, like the domestic accelerator ecosystem, as well as the sort of already despecced American chips and how if you just don't have the bandwidth already available, well, you need to do something about it in order to improve serving at high concurrency or like high interactivity, I guess.

Yeah. Yeah, so the Chinese chips are very interesting. We, in our earlier DeepSeek V4 Pro article, the day zero article, we did talk about the SAN950 chip design and the HBM they use. They don't really call it HBM. It's high, high, high Q. I'm not sure what it stands for, but it's their own version of HBM that they manufacture in China. And of course the bandwidth is lower than the stuff we get, but they're very creative workarounds for their hardware restrictions. And we're actually writing an article on this, which will come out very, very soon. Maybe right after this podcast comes out. I'm not sure when this podcast will come out, but yeah, we have a link in the show notes.

Yeah, basically the trend, I mean, at a high level, the trend forever has been, I mean, China has been restricted in terms of HBM even more than the United States on a chip-to-chip level. So they've just been trying to make the KVs smaller and more efficient, compression KV. And then that leads to having to offload stuff to lower tiers of memory. And I think N-gram is just like a natural extension of that because you're effectively getting Like, as Brian was mentioning, you're getting layers for free in a way, since you don't have to build up the representation and you just have that offloaded on DRAM. And you can, as Brian was saying, you know, you can overlap that with other computations since you have the token ID at the beginning. So yeah, I mean, this is like a general trend of just like Chinese labs are just trying to be more efficient because, you In a way, it's a curse that the US labs have infinite compute and are hoarding HBM and stuff because we don't have, our hands aren't forced in that sense. So yeah, it's been quite interesting. All the innovations feels like they're coming from China.

Yeah. Interesting. Now maybe you guys can talk about the implication on the workload. So obviously the reason to want to store a bunch of KV is driven by many concurrent requests or very large requests. So either long context workloads or high throughput serving, both of which are in demand for anybody that's serving their model for coding agents right now, like agentic coding. So obviously we had this big motivation to try to develop a benchmark that more accurately represents the workload pattern of eugenic coding. And we've applied this for all of the new models and released both the dataset as well as the initial results. It's called AgentX. Well, Cam, can you run us through what is AgentX and what's the implication on the change from like traditional 8K, 1K to using AgentX and then how we were actually able to tease out some of these differences in efficiencies from the model architecture? And clearly demonstrate them using this benchmark?

Yeah. Yeah. Good question. So I think before we were doing AK1K, no prefix caching, just testing at a chip level. Now we moved to AgentX. Basically, this is just, you know, agentic traffic, right? So as you can see here in this image, right, each, the way, you know, when you code with an agentic harness, what's happening at a low level is you send a message. And, you know, the output gets appended and maybe some tool gets called and the output of that tool gets appended to your stream of messages. And then this just keeps getting sent to the model because the model is stateless, right? But as we know, right, the KV cache is just growing like at an insane rate from turn to turn. And, you know, if, you know, when you have million context length models, right, you need to store quite a bit of KV cache. So the idea with AgentX is we're no longer just testing like the chip level stuff. We're testing the entire system, how KD cache moves between chips and prefilled decode disaggregation scenarios and how offloading works, right? So yeah, actually, if you want to pull up inferencex.com, maybe we can take a look at some results.

Yeah, definitely.

I actually, so in the, in the traces that we replay, right, many people are surprised that the cache hit rate, the theoretical cache hit rate is very high, like above 98%, right? So it's just about how much of that 98% can you actually realize in your system? So maybe just click on like, I don't know, B300 VLLM, right? So, um, the points that are, have a halo around them, right? This is with offloading enabled. So you see that this is not very important for the higher interactivity points because really, you know, offloading to a secondary tier of memory is slower, right? And when you're at a higher concurrency and you want higher, uh, higher interactivity, you I'm sorry, I misspoke. When you want a higher interactivity at a lower concurrency, you don't have as much KV to store. So offloading isn't particularly relevant, but when we're at the, you know, 40 to 60 TPS range with extremely high throughput, so this would be something like offline batch inference or even like reinforcement learning rollouts where you have like hundreds of concurrent agentic sessions, offloading is extremely relevant and you see like Maybe like upwards of 20 to 30% cache hit rate just from DRAM instead of HBM. So yeah, this is the idea with AgentX and we really just want to show like what a realistic workload is. And yeah, I don't know, Brian, maybe you want to talk about, I think the coolest thing is the profit calculator.

Yeah. Yeah. So the cost of our— so our benchmark is really quite cool in the fact that we can kind of like, because Segment Analysis also have TCO models, right? Please buy our TCO model. Our TCO model actually calculates like the total cost of ownership of our model, of accelerator chips. In other words, like how much does it cost an hour to run them? And basically like breaks down what this cost entails, like your data center cost, your electricity cost, the chip cost, et cetera. And we kind of give a preview for users of Inferentia X, the final number, which comes down to the one shown right now. And with these costs as well as— so this is the cost basis. And on OpenRouter, you can kind of obtain how much the inference providers are charging. And from InferentiaX, we kind of know how many users, in other words, the concurrency, the throughput. And with all these numbers combined, we can kind of calculate the profit one can make using different chips. So for example, right now, With the best serving, which is GB 300, MBL 72, I believe, you can make about a 40.70% margin, profit margin at those parameters for Kimi K3. And yeah, with DeepSeek V4.1 Flash, you kind of see another trend where with certain parameters or certain settings, you kind of don't make a profit. So I mean, it's really quite interesting seeing this because, I mean, we are going to write up an article soon on the methodology of this, I believe.

I mean, it's quite simple. I mean, go back to KIMI, Jordan. I mean, the greatest takeaway from this is these are actual feasible profit margins that you can get from running open source inference. These things are just money printers right now until the cost of compute sort of normalizes, like to break this down, right? On the current GV300, like with Dynamo and Velo, I'm assuming a 60% or sorry, what are we at? Put it down to 60%. That's probably more so 60% utilization, right? Meaning, yeah, obviously you don't have users fully saturating the system at all times and maybe you have like some nodes down or whatever. So at 60% utilization.

Yeah.

With the 30% license fee that you have to give to Moonshot, you can literally make an almost 50% profit margin serving this on GV300. It's just a money printer, right? So this is assuming like the cash hit rate that we get on AgentX and then looking at the open router data, but this is like very feasible, right?

Um, so yeah, and sorry, Sam, Kim, let's, uh, Let's go back a minute to the AgionX dataset and talk about that a little bit. I think we haven't necessarily introduced that clearly. We talked about how this approximates the use of agentic coding and how this distribution of requests is interesting because the P50 request is still almost 100,000 input tokens. And you keep talking about cache hit ratio. A bunch of people are probably like, how do they know that's real? It's real because we're replaying our own data. So can you talk a little bit about how we collect that and why we think it, why we're pretty confident that like the replays of the data against these endpoints for the AgentX benchmark is actually approximating real-world coding?

Yeah.

I mean, we, we, we have a proxy that intercepts all of the SemiAnalysis employees' agentic coding sessions and We just replayed them, right? So we just take them all and then do some post-processing to get rid of the bad requests. And then we replay them against actual servers running open source inference. And yeah, I mean, this isn't going to be perfectly representative of all agent decoding traffic, but the fact of the matter is like the workload shapes are quite simple, right? You, all harnesses are kind of the same, similar, right? You have Some tasks that you want done. You send that, you prompt the model, the model decides to use some tools, goes and gathers a bunch more context, writes that all to cache, and then does this self-agentic loop for however long it takes for the task to be done. And during this time, the bottom line is most of the context is, is a cache hit. So with that, you need to be quite smart about how you store your KV cache, how you transfer your KV cache, how you route requests to different prefill servers. And, you know, as you can see at the top here, there's, there's a lot of parameters you can set. We're using Nixil for the KV transfer engine. We're using VLLM for, you know, VLLM for KV offloading to CPU. We're using DRAM offloading only. We're using the Dynamo router. And now this is like not necessarily trivial. There are a lot of ways to do this, but yeah, to answer your question, the data is representative because it's just like Straight up, just like coding sessions that we captured. So that's quite cool. You can explore all this on inferencex.com. It's pretty neat.

Yeah.

So when we're saying like, hey, back to what Bryan was saying around the profit estimator per gigawatt, like if we take our real-world traffic, discount it by the utilization rate, and then divide it by real hardware cost numbers, if people are running the latest and greatest chips with the latest and greatest models, It's incredibly profitable right now.

Yes. If you're listening to this podcast and you have like a few million dollars, I would buy like a GB300 rack. No, I'm serious, dude. Like you can ROI, it's just like probably make your money back in like a couple months.

You say that too.

And it's probably better as well, right? For the inference providers because they have their own like private kernels and we're just working with open source stuff. Yeah.

I mean, even if you go back there, I mean, you don't have to go back to page, but even if you put like a 30 or 40% utilization rate, Right? Which I'm sure some of the providers on OpenRouter, no shade, but I'm sure they're getting around there, like pretty shit utilization. It still can be profitable, right? So until the price of compute kind of, uh, I don't know, I'm not a finance guy, but I'm assuming this will normalize, right? Like it's insanely profitable.

Well, I think people question this a lot at the beginning and I want to bring in Alec here to talk a little bit about this. So let me try and tee him up. 2 things have been changing. We talked initially at the start about how the models are getting more efficient and higher quality, which generally means people are able to charge a higher price for the new models because they're higher quality, but then make more money because it's a lower cost to serve them if they're improving in efficiency, especially with all these optimizations for n-grams and KB cache. But then also the hardware is improving. And so there's competition out there. And specifically, another article that you guys wrote was about introducing the TPU, probably the most exciting, looked-forward-to announcement here. Here's a nice picture of the TPU racks. A lot of people wonder, like, you can rent them in GCP, why are they not on InferenceX? Why is everybody talking about NVIDIA versus AMD? But Alec, talk us through the high level, man. Like, what's been up with TPUs? We got some data.

Right, right. So I think, so we had, we ended up testing out TPU v7, which is Google's first real purchasable accelerator through their own cloud. And so we wanted to see, like, do a quick apples-to-apples comparison against v200 and v300, you know, see how's the performance? Is it like feasible and so on? And as you can see from this Pareto curve, The blue is TPU v7, and it's like at the bottom, it's, that's the best part to be. It's the cheapest cost per million tokens, you know, for everything except for like a single point, which is fantastic. And, you know, that depends on, that depends a lot on the, on the TCO for the TPU accelerator. I think for this one, it's specifically the external TPU or external TCO, which is $1.21, I believe. I mean, that's like super good. It's like, and like the performance they were able to drag out of it is fantastic.

Yeah. And this is like, this is early, this is, this is early days for TPU inference performance. So it'll just get cheaper through software improvements.

Right, right. And like, this is the open source version too. So like, this isn't what the, I guess what they be running internally in big inference providers. This is what the public has access to.

Yeah.

And talk about the trend maybe a little bit. Like, obviously TPU is a very different chip than B200, B300, both in terms of like the silicon design, the programming model, how many people have access to it, how open source the community is. But there's a lot of, like you said, apples to apples and then apples to bananas in the article in terms of comparing GPUv7. GPUv7 is maybe the current generation, but v8 is coming pretty quick. We're talking about some open source models that maybe aren't Google's highest priority to optimize for? TPU VLLM is pretty new. What's the trend for the future with TPU? It's probably going to get better from here, right?

Right, right. So I think, as we saw recently, actually, we had, we saw InfraAct and TPU had like the mega kernels and like they got like a huge performance boost out of nowhere. So like, um, I'm, I'm sure we can keep looking forward to seeing that kind of massive performance boost, uh, from them.

Okay. One of the things about the TPU for years is that Google would produce a ton of them, but then they would use them all for internal workloads or for really big customers where they get the whole support of the TPU engineering team. You guys talked a little bit about VLLM plus TPU and how this is in preview. Obviously we're, we're specifically reporting results from one model on one workload and comparing it to B200, B300.

It's a little bit contrived.

What's the broader trend here? I think it seems like we're going to have more TPU over time in the ecosystem. Is that fair? Yes.

Yeah.

So I'll take this and then hand it off to Brian. You're going to see a couple of things. You're going to see mass externalization of TPU as we see more maturity of V7 and then V8 and so on and so forth, meaning that Google is going to stop allocating all of this to internal workloads, they'll still allocate a ton, but they'll sell a bunch of it too. With that, you'll see massive improvements to the open source ecosystem. So for instance, right now, there's no native Torch backend for TPU, right? It basically goes through this whole like different like intermediate representations and like, I don't know, it's really messed up, right? So Torch TPU is going to come out. quite soon. And this is going to be a native Torch backend for TPU. And this is going to make things, the programming model way easier, right? From a software perspective. So you'll see immediate improvements in open source inference. And like, this is just going to be the trend for quite a while, I think. Bryan, you probably have some things to add.

Yeah. I mean, TPU in the past, I mean, Go a bit more deeper into this. Like, TPUs are built around systolic arrays, which prioritizes like data movement. And one problem with it, of course, was that the systolic array was of a fixed size. So with smaller ones, you have to do padding, which is kind of a waste of the compute. And this kind of resulted— and yeah, because TPUs were mainly used for internal stuff, A lot of the models were built around this limitation, whereas open source did not. And this was one limitation or like one thing holding TPUs back for models on Inferentia X. So like the full power of the TPU wasn't really used. And like a year, a few years back, maybe just a year back, everyone was bullish on Google and DeepMind because like they had the most amount of compute. of any lab because of the TPUs. I forgot what the numbers are, maybe something like 3 million TPUs being produced. But yeah, everyone was very bullish because of the amount of TPUs Google had and the amount of research that they did. I mean, if you went to any of the conferences, any of the ML conferences, you'll see a ton of Google DeepMind papers. And the guys there are really quite Correct. There's a saying on X that anyone prominent in ML is either in DeepMind or has been in DeepMind at some point of their career.

Yeah.

So the TPUs actually being externalized is quite a big deal, I believe. And yeah, and it's quite some time since we've seen a real third competitor to AMD and Nvidia. It'll be interesting to see what the next few years are like.

Yeah, I think the first thing I'm looking forward to is moving the, uh, the support in InferenceX from private, uh, a private fork to the actual public repo and getting this stuff running on some persistent clusters there. That's really exciting. The other thing is, um, we, we know that TPUs are being installed in NeoClouds as well. So for the first time, there's gonna be hosting of them outside of GCP, which should open up the ecosystem to have some more customers actually using the TPUs beyond having to consume them through GCP.

People would be happier to not have to go through GCP, man. That's like a turnoff.

We don't need to make this a ClusterMax call, buddy.

I couldn't help myself, Jordan.

Alec, talk to us about the GCP console. What's your favorite, Alec? Give me your top 3 cloud consoles. What do you think?

My top 3, huh? Well, I can't say I've used that many of them, but, uh, I mean, I just kind of leave it to, uh, to the command line, to be honest. Just that I find the command line easier. I don't have to navigate through all the crazy UI.

What do you think Codex or Claude Code's favorite— what do you think their top 3 CLI are for the clouds?

GCP, bro. When you, when you update GCP on the command line, it takes like 14 prompts that you have to say yes to, dude. It's like installs a bunch of shit.

I think that's free. I think software is free, man. I think Claude 3 can do that whole install for you now, or maybe your OpenCLaw. Okay, so moving on from, from TPU. Some other new hardware hit the wires on InferenceX as well, namely Vera Rubin. Let's talk a little bit about the difference between Vera Rubin and Grace Blackwell. So Jensen's sandbagging again, huh?

Certainly. Yeah, he's definitely sandbagging. I mean, look, it's just an incremental improvement, right? So when you see In particular at the spot that people would ask. So this is DeepSeek V4 Pro, which I guess no one uses anymore, but in the, you know, in the range of interactivity where providers would actually be serving, it's just like 3x better. So, I mean, when you think about what that means is you can serve 3x number of users at the same interactivity. So you could charge 3x the amount of users the same price, right? But that's not how you'd price it, but like, I mean, it's just more money, right? More profit. So if you go to the profit estimator, you'll see this reflected. But yeah, I mean, Brian can speak more to the architectural improvements, but yeah, I mean, just more flops, more memory bandwidth. Like it's just better, man.

Okay. One thing I'll lead Brian in, one thing that I've talked about with Vera Rubin on this show before is the fact that it is a little bit more of a minor improvement going from GB300 to VR NVL72 in terms of software support, in terms of like the physical deployment of the system. Because let's say going from H100 or H200 HGX to GB200, GB300, you introduced the ARM CPU, you introduced scale-up NVL switches for the first time, you introduced a new 400-gig network, or 800 for the GB300s. You know, obviously Blackwell SM100 was a whole rev. It had new FP4 data types that the previous one didn't. VeriRibbon seems like more of an incremental change. There's no big change to the scale-up network. There's no new ARM CPU. Everybody should have experience with liquid cooling already deploying it physically. So on one side, one hand, it might be easier to adopt, but on the other hand, you might feel like, oh, I'm not going to get as big of an performance improvement compared to GB300, and I'm maybe less motivated to go for the new stuff. It seems like that's not necessarily the case initially. So high level, like, Brian, maybe you can talk about the initial software experience of porting to Fair Rubin and then, yeah, back up what Cam said, which is like, there's definitely going to be some performance benefits or cost benefits on a per-token basis when comparing to GB300, right?

Yeah. Yeah, I agree. I agree with you that the Vera Rubin step up from the previous generation was not as big as Blackwell was from Hopper. I mean, some interesting stuff regarding Vera Rubin was that, of course, the despec that we talked about earlier, going to 8 high, the reduction in VRAM. And of course, this may be the labs prioritizing memory bandwidth. Because of the scaled network, you're able to like spread your model out and do stuff like YEP, which makes use of more GPU memory bandwidth. So your per-GPU memory capacity isn't as important because your aggregate is what matters, right? And another interesting new thing with Bear Rubin, of course we can't talk too much about the intricacies of Bear Rubin. is that the new LUTD quantization that we mentioned in our previous article introducing Clara Rubin. Yeah, it's very interesting. The first time we see it.

It's quite cool.

Yeah.

So for those that did not read our article, Clara Rubin has a new hardware accelerated path called LUTD in their internal docs. And this was confirmed already, like their PTX I forgot which version already talked about it. And there are like open source Triton branches or forks implementing LUTB. And from a high overview, all it does is a lookup table of quantization. So use, if I'm not wrong, 3-bit keys to look up 8-bit lookup table values. And this like reduces down to about 3.125 bits. If I remember correctly, per weight. And it's quite interesting that NVIDIA gave it a specific hardware accelerated path on Bear Rubin. And right now we don't see any applications of it yet, but it'll be interesting to see what comes out of it. Maybe it's some very specialized use case that the labs have that open source does not have yet, but it'll be very interesting to see what comes of it. Yeah. And thanks Jordan for showing what it is and how it compares to like block scale quantizations like MXFP4 and MVFP4. But it'll be interesting to see what this is used for specifically. And because you can actually emulate this, for those curious, you can actually emulate this in Blackwell to look at like the efficiency. And how well it actually quantizes higher precision weights and can run evals on it and to check the perplexity loss, stuff like that. So yeah, a lot of work coming out of this. Yeah. Clarobin is quite interesting in some regards, although not really that much of a difference here. But the performance does speak for itself, right? We do see crazy increments in performance from Inferentia X. And this is quite early results somehow, and we are getting better results probably very soon. I think before this podcast goes out, we'll probably have more updated results.

Here's what jumps out to me, right? It's a curve. And so obviously it depends which point you pick in the curve to compare to the baseline in order to see what the multiple is. If you take some really contrived point here, it's like infinitely better than just not being able to serve people at 200 tokens per second because now VR and VL72 can do it. But at a midpoint, 100 tokens per second to get 3 times better, and then on the high throughput cases to not be quite at the 3 times better, be more at 1.4. So a 40% improvement makes you question, okay, is this going to be good for high throughput? Interestingly—

This reminds me actually, Sorry, Jordan, but this reminds me of the CPX. I mean, the noise has died down quite a lot on CPX, but Ver Rubin introduces CPX for deseg workloads and splitting up the preview and decode. And right now we don't have CPX on our InfinTex, but yeah, we look forward to getting CPX results soon maybe, and check how CPX improves disaggregated workloads.

Yeah, it motivates people to think about the FLOPS versus bandwidth trade-off at the different ends of the curve and how you can optimize. And so the obvious chart to me is just like the big step up for Rubin as a GPU is the memory bandwidth, which is like, as you go from HBM3 to HBM4, you get almost a 3x improvement in memory bandwidth. And that roughly seems to correspond with this 3x improvement on the big middle parts of the curve where inference is memory bandwidth constrained. To your point, Brian, the fact that this is an early result and these lookup table quantization schemes are not explored, and, you know, some of the other hardware acceleration features in the GPU is not there yet, it just seems like here's us demonstrating that roughly on a per dollar or per watt or per like adjusted dollar-watt basis, Bare Rubin is still going to be more efficient at inference than Blackwell. It's just a better chip. They're not pricing it super aggressively, you know, aggressively meaning they're not price gouging with this chip because they know people are going to buy it. Like, if you're going to get 3 times the performance and the chip doesn't cost 2 times more, like that's the, that's a trade everybody's looking to make now going into next year. And to go back to the profit estimator calculations from earlier, it just seems kind of obvious that people are going to make this calculation going forward and say, yeah, I want the latest and greatest. NVIDIA's done it again. Jensen's sandbagging performance again. There's parts of the curve where they're really memory bandwidth constrained and you want to be able to serve that. Maybe last question for you guys. You do a lot of work with these coding agents. Can you talk about experience using fast tokens? Like a lot of the Implications of these performance curves seem to be related to like people are going to pay a premium for 100+ tokens per second. We have some hot takes on the phone. I won't share mine right now. Can you gimme your initial thoughts on like /fast and /ultrafast and if we're going to see a lot more of this going forward for you guys using it?

Oh, Cam has spicy takes on this, I believe.

Bryan, are you a— let's start, let you start. Are you a super, are you a fast maxi or are you a fast bear? What do you think?

Nah, nah, I don't do fast. Like for my personal use cases, I have like multiple agents running at the same time. So I have like multiple VS— in my personal use case, I use multiple VS Code windows and like multiple agents on each VS code terminal doing basically different tasks. So it doesn't matter to me how fast one agent completes, because when I come back to that agent, it's probably already done given how much time I spend on the rest. But I know Cam has some differing views on ultrafast.

Yeah, I mean, I really, I'm quite bearish on it because I guess my thinking is that As tasks become more agentic, you need to, I mean, really, really the power in these models is that they can call tools very, very well, right? So they can call tools to bring in more context and accomplish things, but the tools take time to run. And if you have a model running on like, you know, Cerebras or something that can serve like 1,000 tokens per second, Right. But then the tool use time takes like 30 seconds, then that speedup in the generation of tokens just is not really realized because you're still bottlenecked by the tool use time. So I guess like if you're just like generating like HTML pages all day, like 100,000 lines of HTML, then yeah, I suppose it would be quite useful.

Wow.

That's a great cue for me to talk about my use of these models.

Yes.

No, no, I'm kidding. Alec, what's your take, man?

What— No, I do have to agree with them. You know, there's times where I need to do something really fast and, or like get an answer real quick or something, gather context and, you know, fast mode's going to be nice for that. But for the vast majority of the time, I also have just multiple agents and I mean, I can just do something else as they're running. I like, I don't need the code immediately. I don't need it to be merged instantaneously in it. You know, it can happen in an hour or 2 even. So, I mean, I just don't use it very much anymore.

I just, I mean, it's so expensive, right? Like, I'm not sure if the ROI is there. So, like, I mean, yeah, especially when you're bottlenecked by, by the tool use time, it's just like really, really, especially like the ultra fast stuff. Like, you know, I'm not paying $300 per million output tokens or Okay.

So let me push this counterfactual. I agree with you for my current usage, but I also didn't forecast myself using AI exactly the way I use it today a year ago or 2 years ago. So I find it hard to predict what it's going to look like a year from now and whether fast tokens aren't going to be useful with any sort of certainty. So let me push you guys to 2 extremes. One is like, do you have the experience where a model is really big? In other words, let's not say really big in this case, but like, or the network's really slow, or the tool use is really slow, or the model just slows down because the endpoint's busy. And then you get really frustrated because something is just not returning something at all. And so can you imagine a world where the models get 10 times bigger, 10 times less efficient, and we just need to speed them up for them to be even useful at all?

I mean, that's a very interesting take. The scaling laws do show that. then the more you scale, the better results you'll get. So it's kind of an interesting approach to just scale. I mean, that's the bitter lesson, right? Just to scale and let compute take over. But I mean, I've been thinking about that quite recently actually. Like, why don't we just scale up and just go very fast to reach a usable token output rate, but like seeing stuff what the industry is actually doing. Like for example, Astra and rumored rumors that 6.1 sold loop transformers, not confirmed. It's really quite interesting, right? To show where the industry is going. Like it doesn't seem to be like just scaling up is the way. There's architectural changes, very interesting innovations. And yeah, and—

Even loop transformers and lots of reasoning is not free in terms of how much computation you need to do, right?

So yeah, yeah, that's true.

Actually, actually good point on the reasoning though. Like if you're on like ultra max, whatever the reasoning, that is obviously just purely to output speed. Like you need to decode very quickly. So that's a good point. I honestly, I think my prediction would be something along the lines of like, you have a harness that can like route to a fast mode for like certain tasks and then like it can batch certain tasks that don't really require low latency, right? Because when you think about it, this is basically what we're describing where like, if you're writing a, you're bootstrapping a React app, then it would be really, really, really useful to have fast tokens. But if you're doing something like Brian's saying, you know, running a performance benchmark and trying to hill climb overnight, it, you know, the benchmarks take like 50 minutes. So if it takes 50 minutes and then you spend $40 decoding a few thousand tokens and then just wait another 50 minutes. That's just a waste of money. So that's my prediction, right? My prediction is like, it'll be very specialized, but it'll still be useful for some things.

Yeah.

How about the other extreme? Let's say the models get really, really small, really, really efficient and still high enough quality. Is there a scenario where you are just bootstrapping a React app or you're using your OpenClaw/Muse/Muse Instinct, Perplexity Computer, or whatever, and you want— have you had the experience of getting instantaneous responses and feeling like, wow, this is really cool?

I mean, that's one interesting direction, but I don't know. Personally, I don't really think there's a chance of us reaching this smaller model state because, I mean, it's been shown in the history of computing that if you make something more more efficient, someone else is going to just lump more stuff into there. CPUs, RAMs have gotten so much better, but then websites have gotten so much more sluggish and so much more inefficient with the libraries. And yeah, I don't think, honestly, I don't think a smaller model, like a world where smaller models are used will really come true, but it'd be interesting if I'm proven wrong. Better for the environment too, as well.

Interesting. Okay. Appreciate you guys going on that little journey with me. I have a suspicion that even when you guys use all of these different agents in parallel and you don't really care about the time it takes to return, that if it was 10 times slower, you would still find a way to get upset. If you went to bed, woke up in the morning, and there still wasn't a response, that wouldn't be good. So there's room for some sort of fast tokens in the future. Maybe it's not all fast, right?

Also, Jordan, people in America can barely afford Big Macs, right? So I don't know if they're going to pay like, you know, $100 per million output tokens. So I don't know, man. But yes, I agree.

Yes.

Like speed matters, right? To some extent, the intel— I always love when Jensen says this. Jensen says this in every keynote, right? Like faster tokens are smarter tokens, right? Obviously he's inclined to say that, right? Because he sells GPUs. But yeah, I mean, like. To some extent that's true. And I mean, think about if you could, you know, basically have instantaneous output, right? That would be extremely useful for, for some cases, uh, 'cause you're just getting the answer right away. But I don't know, man. We'll see. Time will tell. Who knows?

We will see, guys. Okay.

Oh dude, we should talk about TileRT.

Ah, yeah, yeah, yeah. That was a perfect lead-in, man. I was about to wrap this up.

We can wrap it up. I don't think there's—

No, talk about Tyler RT because a lot of people when they saw the 6.1 Astra or sorry, 6.1 SOL ultra-fast mode get released at OpenAI Dev Day, everybody was like, oh, well there's a Cerebras deal with OpenAI that's promising them 750 megawatts over the next few years. Obviously this is Cerebras, but Um, TilerRT is super capable. And if our take on how big these OpenAI models are is correct, and they're using some similar optimization to what we've got on screen here showing TilerRT, well, 200 tokens per second, all the way approaching 500 tokens per second on a decent-sized model approaching trillion parameters like we've got on screen here, it's totally possible just on NVIDIA GPUs, right?

Yeah.

Yeah.

I mean, TileRT, for those who don't know, is basically just a latency optimized inference engine optimized for ultra low latency, right? So I think the takeaway here is that you don't necessarily need super like specialized SRAM architectures, whatever, to get like high, high interactivity. So these are achievable to some extent.

Right?

Bound by the laws of physics, of course, by super optimized inference engines, right? So this has been shown here. There's also an AMD result. And I think going forward, this will become a trend where inference engines will become more specialized for different use cases. We're already kind of seeing this trend. But yeah, TileRT is super cool because yeah, you don't have to Have a new chip. Just take better shots.

Can you talk a little bit about how it works? Obviously there's SRAM on the NVIDIA GPUs. There's lots of different ways that you can configure the serving setup as well as write specialized kernels to actually serve the models. Anything technical we can talk about on that?

I think Bryan can talk about like why it's good at high interactivity, but it's bad at High throughput, right?

Yeah. So basically, like, Taillight just, you can see Taillight as one humongous mega kernel for the entire model. So like the problem at low interactivity is data flow. So what companies like Cerebras and Manova have done is they built data flow machines where the hardware is built for data flow. And for GPUs, this is a bit more difficult. And entirely you can kind of see it. Only batch size 1 is possible right now. And the model releases are quite slow. This is because it takes a lot of software effort to actually get to this mega kernel because, well, the GPUs are not that friendly for this kind of data flow stuff. But it's really quite interesting that GPUs really are general, the fact that they can be used for this and kind of replace the specialized hardware.

But to what extent is this enabled by just having the AI, like an AI agent hill climb one giant mega kernel that literally no one can actually read, right? And then it just works and it's good accuracy and high interactivity, right?

Yeah, but I mean, people have been doing this, that's for sure. But And of course, I personally do believe that kernel development is a very easy task for agents to do because it's very easily verifiable. And we have seen a lot of agents on the leaderboards of GPU-MOD and other kernel competitions. And I don't know, right now this is a very new space. Recently I've seen this kind of mega kernel ideas come out on MLX. There's a repo recently released on MLX, as well as other developments. For example, the TPU mega kernel, which uses the same idea of this mega kernel for the entire model, which is very interesting. Yeah. Jordan, I have a question for you.

Sorry?

Jordan, I have a question for you. If you are, not to name any names, but if you're a chip startup, And your basically entire thesis is like fast tokens. Are you a little bit like concerned right now with the state of, I mean, all these improvements with megakernels like Brian's talking about and Jalapeño, for instance, like, and just the general purpose GPUs just becoming faster at this region of, of interactivity. What do you think?

I'm not concerned because I have a fundamental belief that demand completely outstrips supply and anybody who can produce chips is going to find a customer right now. People just need a lot more chips of all sorts. I think in the long term, all of these chip startups really make or break their business by their ability to deploy things that actually work and to be able to produce a bunch of chips. Like, you just, if you're going to have a really big business, you need to be able to manage your supply chain and actually produce a bunch of chips. Like, you need to be able to get good yield and you need to be able to build servers and just like Build a data center and train people and all of that, which is, you know, in some ways a much harder task than actually making something that performs really well. I think the other thing that I've been clued into is the fact that high interactivity is high throughput if the interactivity is high enough. In other words, you don't need a bunch of batching dynamics just to get a whole bunch of, you know, throughput, even if it's a 1-to-1 ratio between Throughput and interactivity, because if your interactivity goes to 1,000, your throughput's at 1,000. If it goes to 2,000, your throughput's at 2,000. So I'd be driving interactivity as high as I possibly could with a lot of this custom silicon.

But it, it's gonna be like expensive. Like if, if, if you just have like batch size 1 and you have 1,000 throughput at 1,000 TPS, then one user's paying for entire utilization.

You can do batch 1, but if it's really, really fast batch 1, you can just pipeline everything and still get high effective throughput even though there's not A bunch of batching dynamics here. So I don't necessarily feel like fast tokens are by definition inefficient tokens. I think that really depends on the hardware design. And if you're, as Myron says, using, because eggs got expensive, meaning HBM, you're going for caviar, which is the SRAM. So I don't think it's a per calorie thing.

It'll be, it'll be interesting. I mean, another axis that I'm skeptical of is like quite a few of the alternative chip startups, not to name names, do kind of rely on general purpose GPUs for prefill. So like the heterogeneous stuff is cool and all, but if your argument is like, okay, well people are chip constrained and they need more chips, but like if the chip can't actually run efficient inference, like it's just kind of like. You still need NVIDIA GPUs. I don't know.

Yeah, the setup that makes sense to me is some pool of pre-fill decode whereby you oversize one of the pools to be able to serve a non-pre-fill decode disaggregated setup for some of the traffic. In other words, if you're fixing your pre-fill decode ratio ahead of time for the entire data center, I think that's totally bunk because it's just going to The workload pattern's just going to change and you're going to have some wasted inefficient amount of silicon, right? But if the workload pattern changes, you know, for more decode or less decode, and you can move those chips into a, you know, only like a homogeneous broken up setup, I think that makes a lot more sense. In all of these scenarios, if you depend on NVIDIA to be one of your GPU partners, And you're buying GPUs and stuff. Yeah, the business decisions there make a bunch more impact on your success rather than the technical decisions. Because look, NVIDIA acquiring Groq is just obvious justification for all of these chip startups that they're onto something technically. Otherwise, why waste $20 billion acquiring somebody whose chip doesn't work? It obviously works to do something NVIDIA GPUs can't at any price. And so everybody else can do the same thing. But the question is not technically if this is going to be beneficial to anybody or if Nvidia has the demand signals to want to go and build this. It's business. It's like, are you going to be able to build enough of them and deploy enough of them and actually make them work? And I think that's quite a big question when Nvidia's controlling who gets shipments of GPUs and backstopping people who want to buy data centers. build them and stuff like that, right? That's a hard business, hard to get into. Yeah. And I think the other thing is like all of these chip startups have proven at this point, generally speaking, that they've made a chip that they can tape out. I mean, and get like, it works. CPU, Trainium, everybody who's building custom silicon, you know, MTIA, Even Maya, like the hyperscalers have proven you can build general purpose stuff too. It doesn't have to be decode-specific silicon. So why not just build a second chip that's prefill-specific if you want to, you know, go and win, build a prefill and a decode chip, right?

Yeah.

I mean, I think that's kind of what Jalapeño did, right? Or that's what they are claiming at least.

So, I mean, yeah, like The design, like the RTL is not free, but people have proven it's a lot less expensive than it used to be. And the software moat, the quote unquote CUDA software moat is speed to develop these things as opposed to just like people, there's something physically setting Nvidia apart where others cannot reach a certain level of performance. It's like you can if you spend the time and the effort and the money on doing so. And so I expect chip startups to do that. I expect them to build prefill silicon and decode silicon. I expect them to realize the benefits of removing the margin of multiple vendors, if it's NVIDIA or Broadcom or Marvell or whoever they're partnering with. And then I expect them to have a really hard time deploying all of this stuff and producing enough of them for it to actually matter because supply chain and data center operations matters a lot, like I already said. So, um, but I'm excited, man. I'm like, I know it kind of got buried in the middle of this conversation, but TPU on InferenceX is to me the most exciting thing here because it's just like a one-way street. You know, they don't start selling TPUs externally and then just say, ah, we're going to stop and keep it all for ourselves.

Yeah, no, no, it's super exciting. I think I mean, Alec, you've worked with Google. I think you probably can attest to where you think this is going to go. And yeah, it's awesome. I mean, we're going to have Trainium soon as well. We're going to have alternative chip startups. We're going to have everything. That's the goal, right? So I think going forward, like InferenceX, Is just going to be a consolidated place on the internet where you can literally get free information on the performance of chips and real-world inference workloads, um, and compare all of them.

So.

InferenceX chat room soon, Cam? What do you think?

On the website?

InferenceX chat room?

I don't know.

We'll see, man.

We'll see.

Gotta take that out with the boss.

Dylan, what do you think about being a Reddit moderator again, except it's on inferencex.com? Alec, you want to be first moderator? What would your username on InferenceX socials be?

Man, I don't know if I ever want to be a moderator. That sounds like such a pain.

Tip, tip of your thing.

Aren't you kind of a moderator of the GitHub Actions queue for all of the runs that are going on right now? Kind of?

Yes.

Yeah.

We're like kind of gatekeeping.

Well, it's your codex, right? You give 'em the API key and you say, hey, moderate all these drones. Well, actually, that's a good idea. We'll have an AI agent be the moderator. Um, hopefully there's one today.

Yeah.

Yeah.

Hopefully it's more of a Reddit and less of like a—

I wonder if the AI agent will ban Dylan's posting on his own forum or if it'll let it slide. Okay guys. Nice, uh, nice way to end off. Anything you think we missed here?

All right.

Little closing sentiment. Um, as we talked about this, it seems like there was 6 different articles that came out in between podcasts about InferenceX. We talked about n-grams, AgentX, Rubin performance, TPU performance, the, uh, DeepSeekV4 0 to 1. I guess we already talked about that, so that doesn't count.

5.

things.

And Tile RT is the 5th. Good stuff. Hopefully we can get you guys back on before the next 5 articles drop and do this a little bit sooner. Appreciate you taking the time today and thanks everybody for listening.

Thanks, Jordan.

Yeah. Thanks for having us.