Ep. 027 - OpenAI Jalapeño: Better Than Nvidia Blackwell (Accelerators)

Hello everyone, welcome back to Semi Analysis Weekly. Today I'm joined by Brian and Myron. We're gonna talk about the new article we put out on OpenAI Jalapeño, better than Blackwell, their self-designed ASIC, which we compared with Rubin. We analyzed the TCO, we assessed the throughput per megawatt claims, and a little bit more of the details in terms of the microarchitecture, system architecture, how they used it. AI to design it and write kernels, and why the speed from tape-out to actually like first working system with first real benchmarks run on it has been so impressive. So guys, excited to dig in. This is a fun one.

Yeah.

You could say something in response there, Brian.

Sorry, I'm still getting used to this. Are you doing too many of these?

Is this a virtual background that you've got like pink cotton candy background going or what?

Of course. Of course. I think it's the most similar color to what I think OpenAI's blog did. I don't remember off the top of my head, but they got these like nice designs for their blogs. Yeah, but anyway, Jalapeño has been great.

Quite a surprise on the last day of Hot Chips for those attending. One of the most surprising talks, in my opinion.

And yeah, I believe it caught— the performance caught everyone by surprise.

We all knew that the OpenAI chip was in the works and they've announced deals with Broadcom, but a first look at performance results and we— yeah, and it's really looking quite good for OpenAI. A brief summary for those that Didn't read the article, which is kind of unlikely, is that on perf per megawatt basis and perf per TCO basis.

So in other words, how much it costs to actually run the chip?

Actually, Jalapeno actually beats Vera Rubin's July results, and we are comparing against July results as we believe it's kind of fair because. The software is in like a similar state between these 2 points. Yeah. So as you're shown on the picture, this is Jalapeño compared against GB300. So GB300, it beats out of the water completely.

But as we mentioned during our—

in our article, it's a bit unfair to compare it against Blackwell because Blackwell uses HBM3. Well, Jalapeño uses HBM4. So actually Myron can talk about this a bit later on the differences between the HBM, but it's a bit unfair to compare HBM4 against HBM3. So we're comparing it against Rubin instead. And yeah, as in the results show, it does beat Rubin on output token throughput per oil utility megawatt. But then again, these are July results, and Verubin right now is likely much better than July. But jalapeño, of course, will improve as time goes on, as we've shown in another diagram below Jordan. I think we show how jalapeño has improved from 25 days. Yeah, no, 25 days.

Yeah, I'll go grab that one. But Maybe like the key point you were making, if you can explain a little bit more about the benchmarks that they're running, which is DeepSeek R1, 8K, 1K, not really the absolute biggest model and not the most demanding inference workload because it's just random data. It's not agentic workflows. But really quickly, they got this up. Clearly the performance is strong as you're, yes, implying with this new chart, performance is like improving by the week at this point, even by the day. And maybe you can, you can also explain the denominator there, like why they're choosing to divide by the power consumption.

Yeah, actually Jensen brought up this point in Computex 2023. 26, during his keynote, that data centers nowadays are getting power limited and that power is starting becoming the constraint. If you have money, you can always get more servers, more chips, but the constraint is starting to become the data centers power. And like companies have tried getting through this using BTM or behind the meter power, but end of the day, the power, tends to be your, your constraint for building a data center.

Yeah, absolutely. So like if you have 100 megawatts of power, you can only fit so many chips in there. It doesn't really matter if these chips are more expensive on a per megawatt basis. If they're producing more tokens per megawatt, you think you can make more money off of the tokens they produce and you justify the extra expense on the actual chips. And while it's a valid metric for some customers, it may not be valid for others. And so it's, it's not the most typical way. Most typically we see just token throughput per GPU compared, like per package. But in this case, OpenAI doesn't have customers for Jalapeño. They just run it for themselves. And so They don't really care tokens per package. They care about tokens per watt or like megawatt that's going into the system, right?

Yeah.

And actually going off of this topic, someone did mention, I forgot, one of us mentioned that like performance per chip is, at the end of the day, is just an imaginary thing. You can just glue 2 chips together and call it, you double your performance per chip.

So looking at per chip is not really a wrong metric.

It's just, it can be easily misled because you can just, which is what Rubin Ultra and Blackwell Ultra does, if I'm not wrong, you just glue 2 chips together and bing, bing, bong, you got double your throughput per chip. Yeah.

Yeah.

I mean, you could do this if you're Cerebras too, right?

You can just put 3 of the wafers in a rack and then you can put them all on a chart where 3 is better than 1, right?

Yeah.

Or, or yeah, just say your chip is the whole wafer, right?

Yeah.

Yeah. But I mean, it doesn't mean that they don't care about performance per cost, right? Because I'd say that, you know, performance per watt and performance per cost is, they're pretty closely linked, right? I mean, I think the more power the chip consumes, it probably means the chip is more expensive as well, generally. And I think we see that, you know, we did our TCO analysis of what we think OpenAI would pay for a Xiaoping system. And, you know, they still pretty much win or are very close on performance per TCO as well, right?

Yeah, here's that chart on screen. I mean, this one's It's just more interesting because obviously the TCO calculation in terms of like how many tokens you're going to get per dollar depends on that input per dollar. And so we've got some variables on screen there about, you know, what's the total cost per hour to own, for example, a GV300 or to own Jalapeño. And we're assuming that OpenAI is going to pay $2.79 an hour for a GP300. They're buying them themselves. They're running the data centers themselves, for example, in that case. They're not paying the current going NeoCloud prices, which are like $6 an hour for GP300 right now. And put Vera Rubin at $3.61 and then Jalapeño at $1.56. And so let's just compare it per package. Like, Clearly they're gonna, like the thesis of designing a chip in-house is that you want to pay the Broadcom margins only, not the Nvidia margins or the Broadcom plus Google TPU margins or some equivalent there. Right. And that's being borne out on this chart that you can see. All right.

And I guess the, the next part of this is like, where could, like, how could these curves move? I think with Jalapeño, they're actually somewhat sandbagging these results significantly, right?

Yeah. I mean, this is, so the caveats, I guess, on the performance is that I said it earlier that the tests they ran were DeepSeek R1. That's based on the V3 architecture that came out in January of 2025. So it's not the most current DeepSeek V4, but it is a relatively big model, 600 billion total parameters. They ran Kimmie K 2.5, which is 1.5 trillion. So that's an actual big model, or just 1 trillion, sorry, not 1.5. And then they ran GPT-OS-S 120B, their own open source model, relatively small. They had solid performance on all 3, but I mean, solid performance. They're beating Farah Rubin on all 3 today, and they're beating them with single token prediction and no prefill decode disaggregation. So Brian, maybe you can explain single versus multi-token prediction and specifically in these performance claims when compared directly to Vera Rubin, which is using MTP. Um, this is like OpenAI kind of fighting with one hand tied behind their back.

Yep.

Like MTP is a really big optimization for interactivity.

Yeah, of course.

Actually going back on the DeepSeek one, it actually reminded me of an X post I saw like some time ago that someone said the closed lab OpenAI and Anthropic probably have an internal version of MLA and they probably discovered MLA like long before DeepSeek. But it's quite interesting that we were told that the MLA kernels, like OpenAI didn't have any internal MLA kernel.

Yeah.

implementation. So I'm not saying like the open source Triton one, they don't have any internal optimized MLA kernels, which is quite interesting because that, well, that means none of the OpenAI models uses MLA, which is quite interesting to me.

And yeah, and on the 8K, 1K and DeepSeek R1 model choice, we did release an AgentX benchmark case, but the timing was very bad.

So I think we didn't get the OpenAI Jalapeño team to actually run it on AgentX.

But yeah, as Jordan says, measure stuff like prefix cache, which exposes a lot more areas for optimization.

And in our article, we describe like a lot of stuff that is load-bearing that is being tested for AgentX and is starting to show like What's breaking, what's not, and areas for improvement that 8K, 1K didn't properly show. And yeah, so going back to the MTP question, for those unaware, MTP stands for multi-token prediction, and it's a form of speculative decoding where you kind of guess the next future tokens and then verify them in a single forward pass. Because interestingly, the LLM doesn't output just The next token's probability, but the token probability of every single token position before the last token. And we were not given an MTP results. And it's my personal guess that OpenAI has some internal speculative decoding technique that's not MTP and not DSPARK or any open source configs. So didn't give us speculative decoding results because There's no way to actually verify it through open source, which is also quite interesting in my opinion. Means DeepSpark isn't the best that we can do. And DeepSpark actually gives quite a crazy advantage over MTP by doing like guessing all the tokens at once instead of doing MTP, which is just one layer of a model trying to guess future tokens. One by one instead of doing them all at once.

Yeah.

And with, yeah. And we were told that the internal speculative decoding method gives like 3 to 5 times improvement on production models, which, yeah, would just knock the jalapeño versus ferroir comparisons out of the water once again. So you can imagine that graph shifting like 3 to 5 times. Yeah. It's quite insane.

So CUDA mode's gone, Bryan?

I mean, everyone on X is saying, oh, it's just an ASIC. That's what you expect it to do. And yeah, in some sense, that's what we expect an ASIC to do. But the software has come to such a point that like this line doesn't really matter if your software is good enough that you can get a model up like very quickly. What's the difference between a general purpose or like they say GP GPU, but NN ASIC, if software can just bridge this gap.

Yeah. Yeah.

Yeah.

I mean, th— this chip has, I mean, it's tossed up with Veraruna because we haven't seen stuff since they published that note in June or July. So maybe they've, they've had a month extra to develop on these, these chips, but I mean, conceptually, like we've never seen anybody else put out a chart where there's a curve showing they're beating Nvidia on every point of the curve. And like a real test, right? So it's, I mean, it's shocking that they did this. We can talk about the timeline a little bit later, but we keep talking about the point in the curve and I think we maybe haven't explain this in great detail, or we've done it on previous podcasts and people aren't familiar with this. So the— I'm going to put this chart back up on screen and try to explain a Pareto curve here for the purposes of understanding the performance claims made by Jalapeño here. And to do that, we need to explain that the y-axis is how many tokens you can produce per megawatt you're putting into the system. And the y-axis is how fast— or sorry, the x-axis is how fast The tokens appear to each individual user. And so whether you're optimizing for the Y-axis or the X-axis, Jalapeño is beating the GV300 right now on the DeepSeek model, which implies something like if you were to fix the interactivity per user, so everybody sees 100 tokens per second or 50 tokens per second, and I zoom in on like exactly that part of the curve. We're basically seeing that Jalapeño has double the amount of tokens that it can produce per megawatt, implying 2 times more revenue, 2 times more profitability, whatever you want to say, from an inference endpoint serving provider. And then if you look at the far right side of this curve across the x-axis, and you see that at a very low batch size, they can go all the way to 700. tokens per second per user, and you compare that to where the others peak out at 350, this is once again double the performance. And I think the conclusion is that they're basically winning on both sides of the curve. So both fast tokens and cheap tokens, which is It's just so interesting because we've seen so many other companies make claims about how they're going to beat Nvidia and they just pick one of those, right? Groq or Cerebras or any of the other startups that are going to focus on SRAM, like call it a D-Matrix that's coming up with stuff or SambaNova, where we've even seen some results on benchmarks, you know, on the InfraTeks benchmark or something similar to it. They're saying they're going for fast tokens, just decode speed, low batch size. They don't worry about throughput. And then you've got other guys that are worried about throughput, call it AMD just as a simple example, but even TPU or Trainium could be in this bucket of just like accelerators that are going for throughput. And they go, yeah, but we're not going to be able to compete with the other guys at high interactivity. And OpenAI has a chip that can do both for them really well. I mean, at a minimum, This thing is doing really well right now and is going to serve real production tokens for them.

Yeah. And yeah, it's actually quite surprising that, I mean, I was surprised that OpenAI was the first chip that actually, like non-NVIDIA, non-AMD chip that actually appeared on our public InferenceX. We were expecting some Banova or like Cerebras or Even TPU Trinium to be one of the first, but yeah, maybe this really comes, puts into perspective like the difference in, uh, like how this time might be different from the rest.

Like this is the first chip that actually poses a real threat to the CUDA moat because like Inferentia is open source. We welcome, uh, results from anyone. Like if you believe your chip is good, run the results, run the curves, run the benchmarks. Show us what your chips does and we'll gladly put it on our dashboard.

Yeah.

And we'll gladly like write an article about it if it's good. But yeah. And we have extended this offer to like Etched very recently on X. And of course Etched didn't get back to us on that. But yeah, if your chip is good, just run the benchmark, show us the results and we'll let the results talk. Yeah. So we'd love to see more competition from the chips.

Yeah.

Yeah, exactly.

So did you guys see Jensen's response to this? I mean, high level, I think the CUDA moat has been getting drained slowly over the last couple of years. Like top models in the world, like Claude and Gemini are trained without NVIDIA GPUs, you know, TPU, right? Anthropic used lots of Trainium. It's not like you can only use GPUs. But Nvidia's an incredibly valuable company. They're going to keep shipping all these GPUs. And I think there's such a thing as like an Nvidia moat, which includes everything in the supply chain, the whole developer ecosystem, like all of the availability to purchase and support and how you're going to like do the logistics of deploying these data centers and monitoring them over time. I mean, OpenAI's now got to go figure out how to turn on 100 megawatts of these chips, not just like 3 test racks, which is a monumental challenge to get over, as if taping out a chip of this, this, you know, quality is easy, which it's not. Like the next phase will be pretty hard for them as well. And even Cerebras is going through this themselves right now. So I guess the question for me that, well, I think Jensen kind of answered in the style, like I was saying there.

Yeah.

When he was on Mad Money with Jim Cramer, everybody's favorite. And he was basically like, I'm not bothered. And yeah, like, what's like, Myron, what's your take on this? You think, well, okay, what's your take on that? And just the whole, like, just the timeline to go from, yeah, we're tired of buying Nvidia GPUs for everything. We're going to— go build it decision made at OpenAI to actually having a chip that can run an OpenSec benchmark. It's like under 2 years for concept to like real chip in lab and under 9 months to actually get it taped out, right?

Yeah. Yeah. I have several thoughts on this. I guess starting from CUDA eroding, I think a lot of Halide Pinu's success, both in terms of the silicon design as well as bringing up the software, is it's been AI-assisted, right? And I guess somewhat the irony is that this was all done on NVIDIA GPUs, training these models to bring up the capability to the point where— you know, AI is able to, you know, program kernels. Um, and that's been, I think, probably the big shift, um, in terms of, um, making it easier for the labs to, and anyone to adopt, um, you know, alternative systems for their, for their serving stack. I think, you know, one of the big reasons that, uh, Anthropic, you know, has decided to, you know, bring in AMD as one of their hardware providers is because agentic programming allows them to sort of get around the challenges of like using the AMD software stack, for instance. So, you know, somewhat ironically, it's— NVIDIA's hardware has enabled moving off of NVIDIA's hardware. And, you know, in terms of, I guess, you know, Jalapeño, it's such a big surprise because, I mean, we always knew that the team is capable, you know, they have, they've had experience building the other main successful ASIC program in the form of TPU. I think a lot of the Yeah, the hardware team behind this is from, they're former TPU people, as we see in a lot of other, I guess, whether it be AI accelerator startups or other ASIC teams, they tend to come from former people with TPU backgrounds. And so they'll have the experience.

No GPU, for what it's worth, sorry to interrupt, but there's like basically no Chipstart or Second Point 2 where it's a bunch of guys who are ex-NVIDIA. But there's a lot of ex-Google people out there doing stuff, which is interesting.

Yeah, I do wonder sort of why that is the case. But anyway, another topic. So, and I think, you know, I mean, designing an AI accelerator that is competitive with NVIDIA's is not easy, right? It's such a huge market. Of course, everyone wants to try it, but, you know, time and time again, we've seen As you guys have mentioned, we've seen a lot of entrants, but they haven't really been able to do it. So, I think the expectation was that OpenAI would deliver a decent effort with their first generation, and it turns out it was much better than decent. They, as we said, they've already come out with something that's pretty much competitive or better than what the best of What NVIDIA has to offer. So that's surprise number one. And then I think, you know, that also says something probably about other AC programs, like especially, you know, Meta and Microsoft. Is it OpenAI that's really good? Is, you know, are the silicon teams at Meta and Microsoft, do they have skill issues? It's probably a bit of both, right? I think they're the guys that look The worst from this announcement. But going back to, okay, where next? So they designed a chip. Obviously you're scaling up the supply chain to deliver systems en masse. Like you're talking about delivering millions of these chips, thousands of racks, deploying them in data centers, gigawatts of power. That's going to be not easy, but also I think other people have done it successfully. I think the hardest part is really having that system design. And then I think OpenAI has partnered with people who have experience scaling this up, right? Basically, it's Broadcom and Celestica on the system side, and they've had experience doing this with TPUs.

Let me dig in on the comment you made about the difference between like an in-house silicon program that's been going for years and years, like MTIA at Meta or Maya at Microsoft. If we just literally look at the specs of Jalapeño, like it doesn't look super fancy on paper. I mean, it is an HBM4 chip, so it's going to have great HBM bandwidth.

Yeah.

Interesting. It's got lots of FP4 flops, but like still less than half of what Ruben's got on FP4, less HBM capacity and less TDP. So, I mean, like under, like almost 3 times less TDP per chip than Ruben, right? So if you just It's this weird thing where when people are making the bull case for AMD, they go, just look at MI450, right? It's going to have more FP4 flops, it's going to have more HBM capacity, it's going to have more HBM bandwidth, it's going to be more TDP, and therefore it's going to be better. And they never want to look at the benchmarks. They never want to look at the results of the chip. Right. Or the MI355 when it was coming into production for the first time.

Right.

But now you've got an OpenAI chip, which on papers is like objectively worse specs than everything on Ruben, comparable to GB300 on everything except for HBM bandwidth. And yet it's way outperforming GB300 and outperforming Ruben so far. So like, Clearly this is due to the use of, it's due to something about the microarchitecture, which we can get into, and the software. So like, what's your take on what it takes to design a chip now? Because is it purely having access to the latest models and being willing to like YOLO trust them on RTL? And kernels, like this is the barrier.

Yeah, that's a good question. And I don't know the answer to that, but I think, you know, to your point, right? You know, everyone can deliver great, you know, specs on paper, right? You know, when we look at these specs, they're all Peak theoretical. And I think a lot of emphasis on theoretical because I think that some of the FLOPS numbers, no matter how you try to reach them, they're impossible to reach. So it's somewhat determined by the chip companies' marketing teams. But yeah, basically it's There are these FLOPS, but can you actually realize them in an actual workload? Same with bandwidth. I mean, one of the big tenets of OpenAI's design philosophy for Halopeneo is that everyone can deliver raw HBM bandwidth. You just buy the best HBM and then you put more stacks of it. Which, I mean, it's, it's buying HBM itself is not easy these days, but, you know, it's basically not super hard, right? It's not like it doesn't take tremendous design skill. So, but really, you know, what's really throttling a lot of these chips is that they can't realize anything close to the raw HBM bandwidth because there's so many other things in the microarchitecture that stop you from doing that. And I think a lot of that is just whether that be very complicated memory subsystems or just a lot of data movement required. Whereas the Orca AI team has really been able— has focused on the microarchitecture about reduces data movement so that they can realize a lot of this, the massive amount of HBM bandwidth they have, for instance. So I think that's really the main, I guess, skill in all of this and why for the competitors it's been really challenging to, they can deliver all these great specs, but it's really difficult to actually realize them in an actual workload.

Yeah.

Yeah.

And I agree with this, like the fact that you can't actually reach this. I know there's a saying somewhere from someone, I'm not sure who, that the figures are like not numbers that you can reach and are numbers that the manufacturer can guarantee you never exceed. Like those flops, which is like—

I think that's a perfect way to put it. Yeah. Yeah.

Those are numbers that you are guaranteed Not to exceed. And I think in one of the previous articles, I think in the Cerebras articles, we talk about the roofline models of chips compared to NVIDIA. And although NVIDIA's like FLOPS are huge and like crazy, especially the FP4 ones, at the end of the day, they are extremely rightwards on the rooflines in the compute-bound region. And A lot of workloads would rarely even reach those rooflines. So it doesn't matter at the end of the day, the flops, like those flops in the table doesn't really matter because, oops, Jordan dipped again, because the workloads most of the time does not actually hit those rooflines. Yeah. Oh, and Jordan is gone. So maybe, actually, Myron, I have something curious to ask you about, like the HBM. There's been talk about like, Samsung HBM being better than the other HBM and like, Halepino was lucky maybe to, or like maybe it was a decision. I'm not so sure. I'm not a hardware guy. Like what makes Samsung HBM like better or like better quality than the rest?

Yeah.

Yeah. So Samsung for a long time, you know, from the HBM3 and 3E generation, Samsung's HBM was really quite inferior to Hynix, which, you know, who dominated, who still dominates the HBM market share, is the leading supplier for NVIDIA, for instance. And, you know, really 3E, that generation was really bad from Samsung. Part of it was it was built on an inferior process, right? So the, I'd say Samsung, you know, they realized this and they really went all out on, on their HBM4 technology. So the DRAM dies are built on a more advanced 1C process, whereas Hynix and Micron are using a 1B process, the same which is the same process that HBM3 is built on. They're also using, there's the logic-based die in the HBMQ that has NeuronCore 5. And Samsung has built this on an advanced logic process, which is the, you know, SF4 Samsung Foundry 4-nanometer node. Whereas Hynix is using 12-nanometer TSMC. And Micron is still using their own DRAM process. for this base die, even though the bandwidth requirements are significantly higher for HBM4. And that's sort of been, that's why Micron has had some issues with achieving the highest speeds for HBM4. And similar with Qynix, they've had some issues. They've had to say redesign the base die. And this is why— It was only until they've had to delay shipments for HBM4 for NVIDIA's BruteBit, right? So Samsung has ended up being, because of these, I think it seems to be clear that Samsung has actually the best technology for HBM4. This is why The Jalapeño is HBM4. It can do about 15.4 terabytes a second of, you know, HBM bandwidth, which means a 10 gigabit per second pin speed for the HBM4 they've gotten. This is a little bit higher than, say, the 9.6 that we think basically NVIDIA will ship Rubid with. And yeah, I think it probably does end up being that it's because of Samsung being the supplier here. And, you know, the reason is also that whether it's luck or skill, I think we can debate about it, but, you know, I guess traditionally Broadcom's HBM has mostly come from Samsung, right? So I think it's partly sort of that luck. This has hurt Broadcom for 3, but for 4, this has been, I guess, that's turned out pretty good for them.

Yeah, it is very interesting. And yeah, this story of Samsung HBM and like, because if I'm not wrong, SK Hynix invented HBM, right?

Yeah, so SK Hynix, along with AMD, were sort of, they realized that, Very correctly that, you know, memory bandwidth isn't scaling along with, you know, logic performance. So they needed something to really like bring up bandwidth and they came up with this HBM concept. And originally it was for gaming GPUs. So the first product with HBM was on a gaming GPU, on one of the AMD gaming GPUs. And this turned out to be a bit overkill, but thankfully, you know, they had this technology for another very memory bandwidth intensive application, which is AI.

Yeah. Yeah. It's interesting that like SK Hynix and AMD invented HBM, but now like Samsung is doing better in HBM4 and NVIDIA is doing better than AMD. But on the topic of the HBM bandwidth, so bandwidth is a problem, right? And instead of not capacity, is that why like companies are going towards like, if I'm not wrong, 6-high and 4-high instead of 8-high?

So I'd say, yeah, the primary, I mean, it's in the name, high bandwidth memory. The appeal is the bandwidth. Capacity is important, but I think the main sort of benefit is really bandwidth because, you know, there are really cases where you pay for the additional capacity, but you don't, you might not need it. Whereas I think The memory bandwidth, you can always just reuse that into serving tokens faster, right? So, and bandwidth is really the key. And the thing is, like, whether it's an 8-high, 12-high, or 4-high stack, the bandwidth is the same. But because you pay for capacity, and that makes sense because, you know, more capacity, more layers is what adds to the cost of the supplier. You know, the sort of, you pay for the extra capacity, but the dollar per bandwidth gets much worse. So for some companies, if they want to optimize and say, actually, we just want to like get best dollar per bandwidth, then going lower stacks is the right optimization. Of course, you want that balance between capacity and bandwidth, but, you know, I think there is, a philosophy, especially now that HBM is getting much more expensive because we have very limited supply of HBM wafers. So I think the trade-off is starting to look more in favor of going to lower stack heights rather than just increasing them further and further.

Yeah, that's interesting. Yeah, it makes sense with me, especially if like, Increasingly more rack-scale architectures.

Mm-hmm.

Uh, the bit, yeah, like capacity is not becoming as much of an issue anymore.

Yeah, yeah, exactly. Yeah.

And this is, um, I think specifically borne out in the per-watt argument as well, where, um, if you look at the raw HBM bandwidth comparing the specs of the chips, it's like, okay, it's up there with the other ones, but then if you divide by the amount of, uh, power consumed by the chip, The fact that they're getting 15.4 terabytes per second on a chip with a TDP of 700 watts is incredible. Like this bandwidth per watt ratio, like arbitrary units here of 22 is just so much bigger than anything else. Literally double Rubin, even at the max Q, like low power setting option on Rubin to like run it at 1800 watts. That's still Like it's got a little bit more HBM bandwidth, you know, whatever that is, 25% more, 20 versus 15 terabytes per second, but it's literally more than double the power. So I mean, that's where you're— if you can realize that bandwidth with that much power, that's where your token output per watt advantage comes from right there. I guess the big question, of course, Myron, is like B0 is in the fab right now. They should just be able to step up the power and get even more bandwidth from this, right? Or are they at some limit?

I think on the HBM bandwidth, they are probably at limit. I think The, and, you know, yeah, I think that's sort of just the constraint of the memory itself. But the B0, it should do a little more flops at that, you know, same power basically. So that, you know, I think depending on the workload, if there are compute constraints, then that should benefit. For the B0 stepping. The A0 stepping is what's, what the, I guess all the current results are on.

Yeah.

Yeah.

There are many different, we're seeing so many chip startups explore the surface area of possible configurations of chips right now. But if you are optimizing for HBM bandwidth per watt, a metric that seems pretty relevant in LLM inference, this is the design to go with at this point, right? There's nothing else that compares that we've seen specs on, or that are public specs on, let's say. Not giving away too much there. So maybe just to talk about realizing it a little bit more. I don't know, Bryan, do you want to talk a little bit about the software programming model and the microarchitecture, or you want me to talk about that?

I think you're more knowledgeable in the aspect, right? But actually, before I let you answer your own question, another very big interesting point is the role of AI in all of this, right? You can make an argument that OpenAI's biggest advantage is that it's able to access its own Astra models, the new GPT-Astra models before anyone else. So like, and that's like the biggest difference between, I would say, between OpenAI and one of the other new chips companies, the Samanva, Cerebras, et cetera. Yeah, and the question is, how much did, at least my question is, how much did Astra actually contribute to this? Like if you look at it from a differences point of view, like this is the only, one of the only differences between Jalapeño and the other chips. Like, so did Astra really contribute to like most of these performance differences or just a bit? Yeah. But that's a tangent of AI work on development. And yeah, actually, Kimi K3, when Kimi K3 was released, there was a part on the blog about its designing of a chip. I forgot what was it about. Maybe some, yeah, I'm not sure what the architecture was about, but yeah, they did talk about K3 developing a chip. And yeah, I would guess GPT-Extra does have similar capabilities. And I would say to a better aspect or to a higher degree. Yeah. Sorry, back, back to you, Jordan. Yeah. On the microarchitecture.

Yeah.

Well, let, I mean, let me comment on the AI assistance on the architecture, 'cause I think it's, there's 2 ways in which AI clearly assisted the design and then the bring up of the chip, namely design and then bring up. So on the design side, this clearly wasn't Astra because the RTL freeze was in July of last year. So, you know, from February to July of last year, when they claim that AI assistance helped them get an 8% reduction in SIMD area and then a 10% reduction in the matrix engine area during design, I mean, this is pre-GPT-5 that we're talking about. So conceptually, like, the models have to be getting better at RTL In the meantime, but they were already good enough to rapidly accelerate the really tedious human-driven work that is RTL before a tape-out, right? And so I think maybe that's the biggest claim here, which is that I think a lot of people understand you can use these models for kernels and just software engineering because you put it in a Codex harness, you put You put it in a loop, you let it test the thing, and then you just set goal. And like, people have had that experience, they can kind of understand it. But I don't think a lot of people can, like, have had the experience of designing a chip. It's not like traditional software programming. And this was done with an older model. So I think that's like point number one. Now, on the actual bring-up, Yeah, kind of like what I just said, the work is iterative and it's a verifiable domain. So this is kind of exactly what RL should be good at. You should be able to give a model a task of improving the performance of a kernel or getting the kernel to be functionally correct against some verification script, against some test cases, right? And then just let the model rip, let it try. And Dylan made this point on the Dwarkesh podcast that he was on recently, which is that for years, I think now these, the companies that were getting the most value out of using AI were not actually the companies providing AI. Like OpenAI and Anthropic were not profitable for a very long time. And now they're like just turning a profit, but I mean, still running incredibly high-margin businesses, but they're just like realizing the profitability of training these advanced models. Meanwhile, you know, Jane Street's going out there and printing $15 billion in a quarter, clearly using AI for trading or something like that, right? And there's many other companies that are being started based on the use of AI. This is a very clear example of OpenAI Keeping the benefits of having access to a model before everybody else for themselves. They can take about a chip and their competitors can't. And it's a sign of what's to come. I think that's the simplest way to put it. They're going to be able to go into many domains that are tangentially related to software where it's like the model needs to be able to control a computer, But it's not explicitly the thing you're training it for. You build RL environments, you spend enough tokens, you spend enough time on reasoning and enough rollouts, enough attempts at the problem, and you're going to get a good result. That seems to be the lesson here.

Yeah, exactly. And I think Anthropic is also realizing the same thing. They are starting to hire like silicon people that's on the same path as OpenAI, Celebneo. And they are doing a lot of stuff like in the laboratory, they got like LLMs to control microscopes and whatnot recently. And they're going, seems to be going quite long into this like biological sciences field. Yeah. So I really agree with you, Jordan, on the point of like these frontier model companies are realizing What this— oops, lots of interference from Jordan. But yeah, lots of good downstream impacts of having a good model first can have, not just making money from inference revenue.

Yeah.

Lots of interesting developments.

What's your thoughts, dude?

Yeah, I agree. I think the progress at AI labs is only getting fasterized, and that's really because they're using their own models really effectively to drive product innovation much faster. I remember, I think, was it early this year? Anthropic was releasing a new product every week or something. And I think it was like everything was basically on autopilot. They were just using Claude code for everything. So yeah, I think this, I agree with what you guys have said.

Yeah. Yeah. I mean, the cynical view of a program like this for both OpenAI and Anthropic and even Meta and Microsoft is that it's kind of like, a head fake that gets them a discount on the Nvidia GPUs and therefore it pays for itself. You only need to spend a few hundred million dollars on a program to tape out a chip and the year or two to do it to potentially help with the negotiations. And if those negotiations are measured to the tune of hundreds of billions of dollars, Then it pays for itself pretty quickly. But the non-cynical view is that this is a real thing and they're only going to do more of this in the future, right? There's no reason that they're going to be less vertically integrated and less interested in developing chips this time next year. And there's no reason to say that The RTL time from freeze to, or from like initial to freeze to tape out can't go even shorter than 9 months. And I think you just need to think about where to go from there. Maybe the other thing that was kind of interesting here. So like if we go through the architecture, I mean, there's lots to say about it, right? Maybe the quick high level is just like, it looks like a TPU with much smaller systolics, systolic array being the way a TPU has its processing elements laid out. And maybe the criticism of chips like a TPU, Trainium, even some of the ones that are TPU-inspired, let's say like an Etch or a MaddX or something like that, is that when they go with these really big systolic arrays, they can have these weird cliffs where small batch dimensions, like the M dimension in your MNK for a matrix multiplication, gets small, which is what happens when you have lots of experts and you have very few requests, like low concurrency. These skinny gems or skinny matrix multiplications can waste a lot of resources, right? Or even just odd numbers where if you go slightly over 256 or slightly over 128, now you're spending an entire kernel launch on the device side just to run one little skinny gem. All of these like processing elements on the systolic array are not being used. And so therefore it's like inefficient and you don't actually maximize the flops on the chip itself. The trade-off here, of course, is that to get more efficiency with tiling, to reduce the issues with like padding overhead or alignment on the matrix dimensions, is that you just use smaller systolics and that's what they've done here. And so I think that's been really smart. Clearly for efficiency across the curve. The argument the other way is that they're going to miss out on some, I think, power efficiency and data movement efficiency because, well, you have to have more small elements instead of one big element. So what do you do then? And I guess the way that they've solved this is by being really smart about how they place weights in KBs. using synchronization between cores selectively, and then saving the collective network, the NoC, the network on chip that connects the HBM slices and the computing elements together really sparingly. And I mean, the result is like, well, the results speak for itself when you see the performance there, but the results is that this might be both a chip that's simpler to reason about than a GPU, and a chip that's a little bit more flexible.

Yeah.

For some of these weird changing dimensions over time than a TPU. And so, I mean, clearly the guys who have experience using GPUs, which OpenAI has plenty of experience programming GPUs, mixed with the guys who have experience designing a TPU has resulted in a pretty well-balanced system here. Maybe the other thing is that it has an L1 cache, which is quite funny. All of these accelerators do not have L1 caches now. They rely so much on L2, namely SRAM. We hear SRAM all the time. And I mean, the reason why I guess is that the companies designing other accelerators don't want to use a scratchpad. Sorry, they do want to use a software-managed scratchpad, and OpenAI is not using a scratchpad cache here. And so that makes the chip potentially harder to reason about when you think about barrier latencies and where you're going to move data. You have to be able to amortize the data movement by doing work on the CPU itself. But that ties in with what I was saying at the very beginning, which is the second phase of using AI, which is actually programming kernels on, meaning software that runs on the device side. And the To do this, I mean, we haven't really been able to verify this other than scrolling through a 3,000-line file with some of the OpenAI engineers when we went on site with them. It's literally just slop. It's not slop because it performs, but it's literally just AI-generated assembly basically puked out in Gluon, this low-level kernel programming language that they built on top of Triton, which uses this This really interesting programming model. The point is the guys who were scrolling through this code with us, it was kind of clear that they know a whole bunch about hardware, they know a whole bunch about the concepts in the system, and they just have no idea what this MLA kernel that they're showing us for DeepSeek actually does. You can go line by line and it's like, nope, nope, nope. It doesn't matter, right? And the AI understands it, the AI tests it, and you see the results. It produces correct kernels that perform really well. And I think this is just a sign of what's to come again, right? That actual code is not necessarily something that a human has to reason about deeply if the AI knows how to manipulate the data movement and the processing elements on the Hardware that you've given it. Okay, we're kind of running out of time here. We've been going for a while. We've got 3 things that I had in my notes that we want to talk about. They don't do PD disag. Brian, maybe you can rant about that because you spend like all of your time debugging PD disag stuff on InferentX, right?

That's one.

2, we didn't really talk about the system architecture. We can talk a little bit about how they do scale up, scale out. domains. I mean, they don't call it scale-out, but whatever it is, scale-out, the multi-tier scale-up stuff, which is just like a total mess to try to understand. Probably can't communicate it on a podcast. Go read the article. And then the third thing is it's a generalized inference chip. It's not co-designed with their models like they keep saying. And the proof of that is it runs Doom 36 frames per second.

Can I run a processor?

Yeah. Yeah, I remember we were talking to these guys and we were like the angry, disappointed mother who comes in and they show us this groundbreaking chip that's so fast and runs this stuff and they're like, but does it run Agent X?

No.

96% on the test. What 4 questions did you get wrong? Um, anyway, anything you guys feel is left unsaid on the chip? Uh, pretty exciting release, eh?

Yeah, very. Um, I'm, yeah, really excited to see where the roadmap goes next. And also, we're excited to see, I mean, Bryan mentioned this earlier, but Anthropic is building it, or an ASIC. They're hiring it, they're building the team to do it. I think this really sets a pretty high benchmark for Anthropic to meet or beat. But I think, yeah, there's every reason to believe that Here Anthropic could achieve a similar outcome. So very excited to see that as well.

And they're hiring the team now, right? So it's only like, what, a month and a half until the RTL freeze?

Yeah.

And then like, what, 4 or 5 more months for the tape-outs? So like, we should be able to get Anthropic custom, you know, chip tokens After like one, you know, nine-month cycle, right? Like one, one pregnancy term. You don't think?

Yeah.

Anthropics in Anthropics hiring people right now. So it's like the first trimester, and then like the second one, they they get the RTL freeze. So they go to the second trimester, and then like goes off to the fab, and then like tape. out, it comes back. Third trimester, all done, right?

A little bit longer. Yeah.

You guys don't like that one. Okay. We're going to end on one other joke. So Brian, you, you'd like the one about converting energy usage to calories, right? So I want to finish this one. We, We converted to human speech and compared some of this stuff on calories, right? Because a calorie, or how much energy it takes to burn a cubic centimeter of water, I think, is the equivalent of whatever. 1 joule is 0.000239 food calories. So we'll throw this this one up on screen to lead everybody off with a little joke. If you convert all of this efficiency of some of these DeepSeek results that we have on AgentX, we've identified that human speech is roughly 20 times, 22 times more energy efficient than the concurrency 1 B300 configuration that we were testing it against there, right? A human speaks at 3.3 tokens per second, but these batch 1 configurations are Going up at 180 tokens per second, much faster than the human brain can work, consume calories, and produce speech.

So anyway, we're here in— But I think the caveat is that not all human spoken tokens are very high quality. I think that's the main asterisk.

I got to caveat some of my interactions with Claude too then, man. So some of this nonsense that's been spitting back at me recently, I've I've not been too pleased with either. Can't see all the thinking traces anyway, but the Claude version of English it's given me has not been that great. Yeah. Well, if we're comparing the machines on how many calories they're consuming per token and we're comparing it to our speech, we are really in competition with the machines at this point then. Hopefully Jevons paradox continues and everybody that produces chips wants to consume more tokens and produce more And hire more people and everybody gets to come have fun. All right, guys. Good job. Long episode this time. Hope everybody enjoyed the overview of OpenAI Jalapeño. More to come. Last joke before we leave, because I just saw it, the best cover image in a while, I'd say, right? We'll leave that one on screen. Lisa and Jensen enjoying a nice Spicy pot of chana or katsu, whatever the code names were for the trays and the racks and stuff in there. That's the motivation behind that.

Vindaloo.

Vindaloo.

Yeah. Vindaloo, right?

Yeah.

Sorry about that. All right, we'll sign off with that image in everybody's brain who's watching online. Thanks for listening, guys.