Zack Kass examines the global race to build supercomputers, from Nvidia's Taiwan partnership to Elon Musk's Memphis project and South Korea's $570 billion investment. He argues chip scarcity drives innovation and that China may respond by reinventing AI architectures to reduce compute dependence, potentially reshaping the competitive landscape.
Chapters
Talk about this focus on Taiwan and the headlines that we have been getting out from the island this week. And the AI supercomputer that Nvidia wants to build together with the government, TSMC and Hon Hai. What does this say about the epicenter of the tech world in Taiwan?
Well, it certainly suggests that it's going to stay there for a while. It's worth noting that at the same time that they want to build a supercomputer, Elon is saying Xi intends to build a supercomputer in Memphis. Korea has said they're going to build a supercomputer. I think they pledged $570 billion to a supercomputer. And of course Japan has, has signaled the same thing. Taiwan has, as you and I both know, a unique advantage in this because they're manufacturing most of these chips. But it is a signal that everyone sees the importance of building supercompute as a sovereign interest in addition to the energy that's going to be required to run it.
So what about China in terms of the supercomputer space? It doesn't have the access to the most leading edge chips. So can it, Is it going to be able to survive? Is it going to be able to last and be able to compete?
So there are, I think, two things. There are two things to consider here. One is how much can anyone actually achieve the ambitions of supercomputing, right? If Elon says I want to install a million chips in my Memphis facility, how many of those will he actually acquire? If Korea says I want to invest $570 billion in supercompute, how much of that will actually go towards it? That's a big question because you can say you want to spend a bunch of money, but if the resource is already scarce, there's only so much you can. The bigger question, and I think the one that you're asking and a lot of people are asking, is to what extent will a supercomputer actually differentiate and is it of critical sovereign importance actually to have your own compute? And I don't know what the answer is, frankly, if the answer were that obvious, then I think it would be very fair to say, oh well, China is in a lot of trouble because they can't access the chips that are necessary. But two things are also true. One is scarcity breeds innovation. Deep Seq proved this. We didn't think that they had the compute necessary to build a state of the art model. They did. Then they built a state of the art model that required a lot less compute on an inference basis. And so actually sort of recalibrated to us, the rest of the world, what we thought was necessary. And that's why we saw this market capitulation. There is a non zero chance that China, as a result of this scarcity crunch, says, hey, we're going to reinvent the architecture. We're actually going to create a world where we're not in a losing game because the game we're going to change the rules or the nature of the game. And that's. That seems very possible. And especially in a world where we decide AI is of critical importance, they're not going to just sit around and say, well, too bad we're going to lose the race. I would also note, and I think that this is important, it does seem like there is an increasing amount of comfort with the world starting to distribute. And even some of the current chip sanctions seem to be being lifted. And some of these trade negotiations almost certainly include. So there's a good chance that this looks a lot more, we see a lot more parity over the next 12 months.
In terms of timeline, where are we in the AI development? I mean, we only really started actively talking about this space in the last year. And that was when we're talking about training the models and this and that. When do we move on to the next stage?
Are you not impressed? What I have to remind everyone is we've been building statistical machine learning models for most of AI's history. And then we started building with transformers when Google told the world in a paper titled attention is all you need. That changed the world in ways that maybe basically no scientific paper had done for, for computers in quite some time. And we've been building transformers for seven years now, basically. And the transformers have gotten an order of magnitude better and an order of magnitude less expensive. And that is in and of itself incredible. Moore's Law broken, right? The question you're asking is what's the next architecture? What is the next thing? And frankly, it's probably on the horizon. It's not obvious that the transformer architecture is a permanent one, but the neural net has so much juice to squeeze. And we are discovering this both in terms of the performance, the cost and the modality, the amount that a model can do. Right? We're building systems that have human intellectual equivalence. That is now happening. And the real question is, what is the asymptote? What is the limit? And it's probably on a cost basis. What we're probably talking about is not a world where you and I are blown away by something that AI can do in a way that we aren't today. It's actually probably a world where what it can do is so inexpensive, where we're not talking about $570 billion on a supercomputer, we're talking about a million dollars on a supercomputer. Because the relative cost to actually run these models becomes so low. And so anytime someone says, what's next? What I tell them is unmetered intelligence. Today we live in a world where compute is very good and pretty cheap. Tomorrow, we're going to live in a world where compute is very good and very cheap.
