Dylan Patel—GPU Shortage, Nvidia, Semiconductor Supply Chain

There's more GPU flops shipping this year than Nvidia has shipped their entire history for the data center. The moment you even put an order in, even if TSMC started working on it immediately, you would not get a chip out for 3 more months. You know, call it 4, 5, 6 months between, you know, when an order is placed and you can actually have it installed in your data center. TSMC actually does have slack capacity for making the chips. They don't have slack capacity for putting the chips together with the memory.

So I'm here with Dylan Patel. A lot of people at ICML have recommended I talk to you about your blog, SemiAnalysis. What's your background in hardware? Like, how did you get into this?

I started consulting on the side of my job, hush-hush, in like 2017, and then I went full-time in 2020. When I got bored in 2021, I started a newsletter on Substack, and then that newsletter's done really well.

SemiAnalysis, right?

Yeah, SemiAnalysis. And so we actually post mostly about hardware, and a lot of times AI hardware as well, but a lot of times hardware. But from time to time, turns out systems people know a lot about architecture of what's running on their system, right? So that's sort of like the backend into like, and trying to figure out what infrastructure is needed, how much networking do I need versus compute versus memory bandwidth? What's the order of operations and how are things scaling? So how does that influence hardware design? And that's important for like my business itself, right? Not the newsletter, but the business, right? Like is understanding these sorts of things and the cost of running these models. What's your model? It's consulting for semiconductor industry, right? So that's the business. And I'd been doing that before I started the newsletter and the newsletter's fun, right? And it's grown a lot and has a lot of people reading it. So that's fun too.

What's the last news in the past month? So I've heard a lot of shortage in GPUs. It's harder and harder to get GPUs in clusters like A100 or H100. Do you have any insights on this? What's the demand and offer and this kind of—

Yeah, I mean, since November, people have been wanting GPUs and that's accelerated this money has flooded in, right? Meta sort of started investing heavily in GPUs and more so like August last year, but they really, really accelerated in like January, February.

Was there anything that caused them to accelerate?

Well, of course, there's the ChatGPT moment partially, but also there's a lot of stuff around the recommendation networks and the infrastructure they were using there that they needed to update. And so there's multiple things, but you amalgamate that across every company, right? Okay, that's when Meta started investing. Well, Google started investing at this time. And then of course they don't buy so many GPUs, they mostly buy TPUs, but That production capacity is actually very similar from TSMC, you know, CoWoS and 5 nanometer for 2PV5, all that sort of stuff.

So you're saying that then TSMC was bottlenecked by not having enough workers and the price went up?

It's not really TSMC's workers, right? I mean, the thing about hardware is so different than software is that it takes, the moment you even put an order in, even if TSMC started working on it immediately, you would not get a chip out for 3 more months. And that's if the chip was already designed and you already submitted the design with TSMC, and that's a multi-year process, right? And then taped out and all the bugs were fixed and et cetera, et cetera, et cetera. So it's a multi-year process for even just getting a chip that you've designed or started to design into production. And even once it's in production, if you wanna increase orders, it takes 3+ months to just get the chip back from TSMC, let alone put it into a server and then get that server installed in a data center. So it's a multi-month process, right? And where software is like kind of, you think about it being like snap of the finger, right? Especially like a lot of the open source stuff we see is like, yeah, some guy was working all night and he uploaded this and now another person's working all night and actually, oh, now another team of people's working all night. And it's like everyone's night is different. So then they all worked all night. And so it's like you see things iterate really rapidly, but in hardware, it takes, call it 4, 5, 6 months between when an order is placed and you can actually have it installed in your data center. if it got worked on immediately. Now the other aspect is like, yeah, there's some slack capacity, right? I mean, you think about manufacturing, you make a massive capital investment of billions and billions and tens of billions of dollars, right? And then you start running the production, but then it's like, oh, well, I want more chips of this kind than you have production of that kind, right? So it's like, okay, well then now I need to expand production. So that's the thing that's going on. And we've done a lot of work in that, right? The equipment supply chain, right? There's about 30 to 40 different equipment suppliers in the supply chain for not just making the chips, but actually just packaging them together, right? And that's one aspect that we've been following, and that's actually one of the biggest bottlenecks is because TSMC actually does have slack capacity for making the chips. They don't have slack capacity for putting the chips together with the memory, with the HBM memory. So that's actually the biggest bottleneck.

Where does the memory come from?

The memory comes from SK Hynix and Samsung, Korean companies. They dominate the memory market, SK Hynix especially. And so that is another aspect like increasing production for, right? It's a different part of the supply chain. And there's many parts of the supply chain, right? Like power supplies and cables and optical cables and all these sorts of things need to harmoniously come together. And each of those has a different supply chain. with sometimes different, a lot of times different equipment and different lead times. And hey, this takes 3 months and hey, I have excess supply here and so inventory so I can sell you some more.

Sure.

It's like everyone has to balance their supply chain and move up. But the long and short of it is that there's actually a humongous increase in GPU capacity, right? There's more GPU FLOPS shipping this year than there have that Nvidia shipped their entire history for the data center, right? That's how much they're shipping this year more. Part of that's off the back of the H100 being, call it 4 to 5 times more FLOPS, but also part of that's off the back of like there's more units too. But it's just that so many people want more GPUs, right? And if you go to like Bay Area companies, right? Like you talk to like all these Bay Area companies, Inflection and Meta and all these, and Google, and sorry, OpenAI and everybody's bragging about how many GPUs they have. And it's really cute. And it makes sense, right? Researchers want GPUs to run their experiments. And so it's funny.

Yeah. He's comparing how much money you make is how many GPUs you have. Do you think the increased demand in GPUs and the stock of NVIDIA going up, et cetera, all this money in the market will turn the whole process more efficient? There will be competitors producing more things and then at the end, or do you think in the next 3 years it's still going to be NVIDIA, TSMC, and all these, the same players being bottlenecked by some small things?

No, I mean, the production is increasing rapidly. NVIDIA is going to ship over a million H100s plus A100s this year. And that's a lot of GPUs. I mean, you think about how many GPUs GPT-4 was trained on and it's like, well, actually, yeah. And then there's multiple labs that are going to have enough GPUs to train something. Not to say they have the skillset, But, you know, to say they have enough GPU horsepower and presumably they have enough skill, they have a lot of the skill set because it's a lot of people from Google and OpenAI who have left and created these companies. But, you know, I do believe that there's going to be, you know, 5 to 7 companies that will have a GPT-4 size model at least, right, in terms of total FLOPS. Right. It could— it's going to have a slightly different architecture, maybe, maybe fewer parameters, more tokens or vice versa. Right. You know, there's, there's a lot of innovations to do on the software side as well. But there's like 5 to 7 companies who will have a model that is that many GPU hours or compute flops, GPU flops in the next year of the quality of GPT-4. So it's like, or of the size of, right? And then quality is beholden to a lot of other things.

I'm not sure how much you believe we could get human-level AI in this decade, but assuming we go these levels by just scaling models up. We can just scale the models 1,000 times and we get something close to human level across many different tasks. Do you think hardware will be the bottleneck? We'll have enough money to do those kinds of things, but we just don't have the supply chain to do it?

I think there's so many GPUs being shipped this year and next year. I look at the supply chain and the amount of GPUs that are going to get shipped next year is freaking crazy. And at least that's what the supply chain is preparing to do. Now, of course, is there demand for it? But I think, yes, we can just scale another 1,000x from GPT-4, right? And that is something humanity can easily do. Does that deliver? I mean, we've seen sort of, I like to joke, transformers are the most capitalist thing that's ever been invented because you just throw more money at it and it gets better. Log linear scaling is perfect, right? You look at these scaling laws, it's like, yeah, it actually just keeps getting better. So I don't know if like, do we run out of data? Well, then we just go multimodal and that's more expensive on compute. But like, I don't see why multiple companies, OpenAI, Meta, and multiple other companies, why don't they have, why isn't their next model trained, or next mega model, maybe in the next year or 2, gets trained on a system that's 125,000 H100s, right? Which are 3x faster. So it's like, okay, well then that's That's a 100x or 10x, whatever. And maybe they train it for longer. I think you can continue to scale and hardware will get better and the ability to build a larger and larger supercomputer because the economic benefit from these massive models is getting better. So I think people won't be bottlenecked by hardware, more so about the ingenuity of putting them together and creating software that can scale and all that sort of stuff and work on. I don't think people are going to use dense models. That's stupid.

But yeah, it's more like the demand is not linear. If I buy a million GPUs from NVIDIA, they will probably not want because they want to keep some of their offer to other players. One company can't just buy all the GPUs from NVIDIA.

Well, yeah, they don't allow that, of course.

Yeah, they don't allow that, of course. So you can only buy, let's say, 10% or something without the price going up too much.

Well, I don't think they're raising the prices. They're just literally saying we don't have allocation. And if you want, more GPUs, then wait till we have built more, right? Like, you know, they're going to build about 400,000 GPUs and sell about 400,000 H100s in Q3, right? And like, okay, like, you know, plausibly Meta is going to buy like 30,000, 40,000 of those, right? And OpenAI is going to buy like 30,000 or 40,000 of those through Microsoft. Then Microsoft for their cloud will buy another 10,000, 20,000, right? But like, you know, I'm sure they want more, but like, you know, it's like, well, Nvidia is going down their whole customer list of the big customers like, yeah, I want to give this many to this person, this person, this person, this person, and next quarter you can buy more. And I think that's sort of how they're playing the game and making sure nobody gets all the GPUs because they want AI to sort of be democratized because, or not democratized, but they want to have many customers to sell their GPUs to. So it's actually in their success for like, I mean, this is why they invested in Inflection, right? It's like, oh, I want another AI lab who's buying GPUs long-term.

Who invested in Inflection?

Nvidia did. Many other people did as well, but Nvidia did.

Yeah.

And like, it was like, oh, well, who's these random, this random startup? Like, do I want to get them 22,000 GPUs this year? Well, actually, yeah, it's better to, you know, take 5,000 away from Meta and 5,000 away from Microsoft and 5,000 away from Amazon and give them to these guys because that creates another competitor.

Right.

And so I think they are also playing that game.

I think that they also like raised like something like more than $1 billion or something.

Yeah, of course. And Nvidia was one of the investors.

Yeah. I think I ran out of questions. Do you have like any last message to the Twitter or YouTube audience? Something that people that you would pick to people and you want to like have them learn or something?

Yeah, I mean, if you're just interested in learning more about semiconductors, I think they are the most complex thing in the world. AI is awesome and I love AI, but semiconductors are infinitesimally complex. There's millions of people working on things and each person is in such a niche of the industry. It's amazing how much you don't understand even after having worked in them for a long time. So yeah, I'd say, hey, figure out working on semiconductors. It doesn't necessarily need to be chip design. It can be aspects of production. It could be aspects of design, manufacturing, and many things.

But yeah, so check out this guy's blog, SemiAnalysis. It's great.

Cool.

And hire him for consulting.