Advanced AI Accelerators and Processors with Andrew Feldman of Cerebras Systems

There was a, a paper, it was published by Argonne National Labs, and it compared us to a 2,000-node A100 cluster called Polaris. It was to use a large NLP network to predict mutations in the COVID virus. What they did is they put the entire genome, 30,000 base pairs, in the attention window, in the long sequence. Window. Then they ran GPT-style networks at 250 million, at 2.5 billion, and 25 billion parameters. On this network, we were 832 times faster than a GPU. So there was an example where 4, 8, and 16 of our machines could do things that thousands of GPUs couldn't do.

You're listening to Gradient Descent, a show about machine learning in the real world, and I'm your host, Lukas Biewald. Andrew Feldman is co-founder and the CEO of Cerebras Systems, where he builds giant computers dedicated to training large machine learning models. We really get into how his computers work, and I hope you learn as much from this as I did. All right, well, I guess before, you know, before I get into it, for people that don't know, maybe you could introduce your company and tell us how you started it.

First, Lukas, thanks for inviting me to chat. We appreciate it. My name's Andrew Feldman. I'm one of the founders and I'm the CEO at Cerebras Systems. Cerebras builds big, fast computers, big, fast accelerators for AI work. And we founded in early 2016, a very unusual approach. We chose to build a very, very big chip. In fact, the largest chip in the history of chip building. It's about the size of a dinner plate. When most people built chips the size of postage stamps. The results were that we were able to put this chip into a system and write software for it that takes customers' TensorFlow or, or PyTorch and runs it blisteringly fast. And we now have customers in the US and in Asia and in Europe and recently announced that we put a collection of these machines together to build a a giant AI supercomputer that's being used by multiple different customers. So what we do is we try to make AI compute extraordinarily fast.

And so what's the advantages of having it at one gigantic chip?

First, you know, as you remember, all little chips began as big chips, right? So the chips are made, by dicing a wafer. And the wafer is, for most of the chips you and I deal with, are again at 300mm in diameter. And the chips are sort of punched out like your mother would punch out cookies from cookie dough. What's interesting is then for large AI work, we have to try and stitch them back together again, right? One chip isn't enough. And so we then expend tremendous effort trying to tie these chips that earlier in the manufacturing process we cut up in discrete elements. We try to reassemble them in the form of a cluster. And it turns out that that's rather bad strategy. That on a chip, on a processor, one communicates very, very quickly and for very, very little power, femtojoules per bit. When you have to leave a chip's boundary, you have to leave, you You have to do all this work, this encoding, and you have to push it out either with SerDes or over some other technology. It travels sometimes over an interposer, over copper wire that's embedded in a motherboard, or sometimes out an optical switch. That's really slow, thousands of times slower. So if you can keep the traffic on your chip, you're thousands of times faster at a thousandth of the power, girl. And so by building a big chip, we were able to keep a huge amount of the work that usually leaves chip boundaries and slows down training. We were able to keep it on the chip and do work much faster at much lower power.

And I guess I have so many questions on that topic, but before we get into it, what are maybe the downsides of having one gigantic chip? Like, why doesn't everyone do it that way?

Well, uh, ironically, uh, lots of people had tried and everybody had failed previously. Um, it's hard in the chip-making process. Uh, there are naturally occurring flaws in the wafer, and, uh, these flaws were like your mother rolled out cookie dough into a round circle that was 300 millimeters in diameter and threw up a handful of M&M's. They landed in your cookie dough. Imagine you're allergic to chocolate and you couldn't eat any of the cookies that had an M&M in them. Now, historically, the answer to that was to say, we're going to bake the cookie, we're going to dice into cookies, and we're going to throw out the ones with M&M. We're going to throw out the ones with flaws or sell them for less. And for most people, this meant in their mind that it as your chip got bigger, the yield dropped. And that means you had to, as the cookie your mother cut out got bigger, the probability that it hit an M&M was larger. And if you had to throw out the cookie, you had to throw out more cookie dough, right? That was really bad. We invented a technique that rather than relying on perfect cookies, cookie dough without any M&Ms. We invented a technique to withstand the flaws and that hadn't been done before. And the result was we didn't need perfect wafers. We didn't need this thing that nobody could ever get. And so we used a different type of thinking. And ironically, it's the same thinking that that Google put to work years ago. They looked at servers and said, people are building these extremely expensive servers that are highly reliable. And they're, what if instead we use lots of cheap servers and when we failed, we just routed around them, right? We could build this at vastly lower cost and achieve better reliability. And so that same approach of recognizing there are gonna be failures, they're unavoidable, and inventing techniques to withstand or that are robust to those failures. Ours was to route around, to have some redundancy that enabled us to be the first company ever to yield a chip this large. And it's not just a little bit larger. I mean, it's 56 times larger than the largest other chip ever built. I mean, to give you an idea, we have, you know, about 2.6 trillion transistors, and the largest GPU has, you know, 50 billion, right? So we're 2.5 trillion transistors larger.

Okay, so Andrew Feldman, um, I'm curious what parts of this you had in mind when you started the company? And that was, that was back in 2016, right? Like, was it with— you see this, like, I just want to make like a gigantic chip, or was it like, you know, I want to make something for, you know, machine learning workloads, or what? Yeah, what was it?

I think we saw the, the rise of, of AI, and we got it wrong 100 ways, but what we got right was that in 2016, AI was utterly unimportant in the economy and that it would be enormously important going right? We, and we, we saw that, and we saw that, that deep learning in particular, uh, is constructed in a way founded on sparse linear algebra, and that the, the GPU wasn't the perfect machine for that, that the GPU was a machine that was built for a different type of workload. And what we saw was an opportunity to build a machine that was better suited that was optimized in every way for this one thing, this sort of deep learning workload. And you know, ironically, that was the history of the GPU when they started, right? They saw an opportunity to build a machine optimized for graphics, right? And that was exactly the, the strategy 25 years ago. And over those 25 years, they tuned and they optimized and they optimized. but the graphics market stalls and it isn't growing like it was. So then they looked around for other things. We saw this new work that was not particularly well suited for the GPU, was very poorly suited for a CPU. And we thought that we could build a machine that was perfect for it. And we knew, we didn't know that we would go to wafer scale, but we knew that what is hard about this work from a computer architecture perspective isn't the calculations, it's the moving of information. And that's really what computers do. They calculate, they store, and they move information. That's it. And what's hard in this, for this problem, isn't the calculation. Mostly we do multiplies and accumulates, right? Most of this work is, matrix, linear algebra, that's not hard for a machine. What's hard for most existing machines is you gotta move those results in and out of memory to others, to other processors that are in another rack or in another row or, and that movement we knew was a problem that had to be solved in optimizing the architecture. But that's what we worked on and that's who why we are special sauce is around the moving of data and the keeping it in one location on the way.

And you mentioned the word sparse. Like, is there something about the sparsity that you also handle? Because I— my understanding of, of a GPU is it wouldn't really handle sparse data in any special way. It would just like leave all the zeros in place, right?

So, um, there, there are trade-offs we all make in, in the design of hardware, one of the trade-offs the GPU makes is it has relatively little memory bandwidth. So it wants, what it wants to do is move a fair bit of data in, work on it, and move the results out. What it doesn't want to do is move things back and forth a lot. To do that, what they do is they sort of are founded on something called sort of BLAS 3 computations. They move data in and do a giant matrix multiply. And they do that as a technique to overcome the weakness of limited memory bandwidth. We have tens of thousands of times more memory bandwidth than they do. And we don't have to move in huge blocks of data at a time. So we can do not just matrix by matrix, but matrix by vector or even scalar times vector, ax y. And if you can have such fine-grained control, you never need to multiply by zero. You can block them out. And so a way to think about that is if you only move pallets of boxes and sometimes a box is empty, you're still going to put it on the pallet, you're still going to drive it across the country. But if you have the ability to weigh every single box before you do anything with it, before you ship it, then you never ship empty boxes, right? And that's why we have the advantage in that ability to get to the fine-grained control that says this is a zero. Multiplying by zero is boneheaded. It's about as boneheaded a thing as you can do in compute. It takes time and power, gives you no new information. Because we can have this massive memory bandwidth, it allows us to have this fine-grained control that says we're going to knock out that zero, not multiply by it. Whereas the GPU brings all the data in, zeros included, multiplies it, takes it all out.

So is your natural representation then sparse matrices or sparse tensors typically?

We can run sparse or dense, but when provided with sparse, we harvest the sparsity by meaning we get a performance boost because we're not wasting time multiplying by zero. Published a series of blogs at NeurIPS showing that we could train models that were 90% sparse to state-of-the-art accuracy, including GPT models. And they took far fewer flops to do it and could be done in much less time.

So even at 90% sparsity, the sparse math is faster? Oh yeah. Wow. I mean, that's still pretty dense.

90% sparse means 90% of the weights are zeros.

Right.

And so what you're left with is 10% of the weights. It is much faster.

And so I guess, how should I think about the chip that you're making? Is it sort of like you've printed a whole bunch of kind of smaller chips all in like a grid and they all talk to each other? What's the best way to think about what you make?

Yes. This is the, the chip. Um, you know, so it's— see, it sounds just how big it is.

Uh-huh. Love it.

My wife says I'm like, like a kid with a— his dirt bike that he got for Christmas, and he rides it into his room at night. He's always got one nearby him.

As you should be.

It only costs $250 million to make the first one. So, uh-huh. Yes, it is constructed of smaller flashes of steps. And that's because the equipment at the fab, and that equipment's made by ASML and our fab's TSMC, but the steppers can only do about 830, 840 square millimeters. And so what we invented were techniques to overlap the stamping. And so when you were done, even though you're using an 850 square millimeter plus or minus stamper, what you've got is the behavior of sort of one giant chip, not a bunch of little chips.

And I guess what did the instructions even look like for this? I mean, I guess you have a whole bunch of things running in in parallel, are they all on the same timer?

They are. There's one clock in any one-clock distribution.

And does the clock, does the clock, is it able to actually, because these are all, you're getting on the order of, is it, so it's the speed of light is what, it's like a, like a, um, in one clock cycle, can you go from one edge of your, your ear tip to the other?

No.

So how do you even keep the, how do you even keep the clock synchronized? What does that even mean to be synchronized?

And that's actually a really interesting question. And there's certainly people on our team who are, Sean and Gary and Michael and JP and Michael Morrison, who are far more knowledgeable than I am.

Sure. And I wouldn't say I'm an expert, so feel free to make my coordinates.

Machine has 850,000 independent processors, independent cores on it. Each core has its own memory. And we call that a tile or a programmable element. Each core has links to its neighbors, east and north and south and west, as well as redundant links that skip a neighbor to northeast, southeast, northwest, southwest. It is going to take a few clock cycles to move across the chip. There is skew. right? So everybody gets the same clock, but there is a delay in its propagation, but it's known 'cause we know where everything is. And so that information can be used, corrected for, and taken advantage of. So all, it's a synchronous design. And so there's only one clock domain.

And did you have to come up with your own instruction set for this?

We did, yeah, come up with our own instruction set. We wanted an instruction set that didn't have a bunch of stuff that you don't need, right? Well, in computer architecture, one of the first steps to decide what you're gonna be bad at, right? What you're not gonna do, right? We're not gonna be a database machine. We're not gonna, this is work that we're not gonna do. And instructions that make that type of thing faster or better, we don't want to carry with us. And so by just putting a line around what you're not going to be good at, really forces you to focus on what you are going to be good at. And how can you, how can you focus your energy on those things you're going to be really, really good at? So how do you treat tensors as first-class citizens? How do you have a technique to add randomness in hardware? How do you, what are the things that that can help this workload and bring nothing else forward, right? And that's how you're efficient is when you are using your instructions for exactly the type of work that you're asking the machine to do.

And I guess what are the workloads where you really, where your machine really shines? Like what, I don't know, feel free to brag a little bit, but what are like, what's the, like what am I doing where I'm like, wow, this thing's just so much better than anything else I could do?

There was a paper published at the Supercomputer Show. It was published by Argonne National Labs and it compared us to a 2,000-node A100 cluster called Polaris. And the problem was really interesting. It was to use a large NLP network, a GPT-J style network to predict mutations in the COVID virus. How cool is that? What they did is they put the entire genome 30,000 base pairs. And when encoded, it was a sequence length of 10,240 in the attention window, in the long sequence window. Then they ran GPT-style networks at 250 million, at 2.5 billion, and at 25 billion parameters. And on this network, we were 832 times faster than a GPU for the 250 million parameter version. The 2,000-node cluster couldn't do big parameter size and long attention sequences. It barfed. It ran out of memory. And we ran it at 2.5 billion and at 25 billion. Now this paper won the most prestigious award in academic computing, the Gordon Bell Prize. So there was an example where 4, 8, and 16 of our machines could do things that thousands of GPUs couldn't do.

And so I guess what was unusual about this situation, right? Because I think these networks, GPT-J, is typically trained these, these A100s. So, and then typically, the sequence loss input.

Yeah, that's right. And what happens is that puts pressure on the calculation done in the attention head, right? You're doing analysis that you're understanding each gene in this case within the context of the entire genome. If you were doing it in a more traditional language sense, you'd be analyzing this word within the context of a page or a chapter or an entire play, um, entire book. And so, uh, that combination of very long sequence lengths, which is the, the relevance window, the attention window, plus big parameters, was brutally memory, uh, intensive and, and caused the GPUs to barf.

Okay, this is a dumb question, so I feel a little bit ashamed to just sort of say it, but I'm an occasional dabbler in training machine learning models. And you haven't sent me one of your systems, but if you did, I would happily use it and endorse it. I can be bought with a lot of compute.

I appreciate people who are honest about their saleability.

I would be so pumped if you wanted to give me one. But I have used TPUs and I've used a lot of GPUs. You know, my experience with TPUs is they do seem like they're, you know, faster in some cases, but I feel like every detail is just like painful. Like when I, you know, when I try to use them, like the, you know, the error messages are kind of weird and I'm not like a real hardware guy. And so it's just, it always just feels like it's just frustrating. Like when I deal with them and there's also, it feels like there's kind of this lag between, you know, when my, you know, when I start my training and then there's like this kind of like long pause and then I get like a weird error message I don't understand. And then I have to kind of go back through that. That leap, I guess, like what's, I mean, is it sort of, I would imagine, my worry with your system, if you sent me one, if I wasn't, you know, on your payroll, would be that it's gonna be like even more like that, where it's just like really hard to like experiment and mess around with it. But tell me, what do you think about that? I think that's a real challenge with TPUs.

And I think we've worked really hard to make it painless. I think in all honesty, we probably have not succeeded completely, right? There's gonna be some things that are a little bit different, but I think we, you shouldn't have some sort of periods of concern once you hit compile, whether it's A, compiled or B, crashed and you're not quite sure yet. And I know people have complained to me about that on the TPUs. We work really hard to make the error messages reasonable. We've worked really hard to make the documentation sort of thoughtful and to comply with PyTorch semantics. And you should be able to take your PyTorch and compile in straightforward manner. Is it, is that true for every network you can think of? No. But for the large NLP networks, you know, GPT-2, GPT-3, and starting at 1.3, 6.7, 13, 20 billion, GPT-Js, uh, T5, um, BERT, for, for the, uh, big NLP networks you care about, it should be pretty darn close to push button.

Now again, I'm, I'm not, you know, the world's expert on this stuff, but, um, I feel like one of the trade-offs PyTorch made was to sort of allow you to execute even like arbitrary Python in the middle of a training run, which I love.

We wish they would take that away.

I know, that's what I was gonna ask. And do you sort of subset like what you can do in PyTorch to make sure without causing some problems?

We're gonna kick that out to a host processor for sure. Running, that was a terrible decision. And for everybody in AI, that was a terrible decision. To let you to sort of put in arbitrary Python code. That's problematic. You know, we spoke to the leadership at Facebook. They'd been trying to find a way to allow quick and easy compilation of that. It's a known challenge for everybody who's building accelerators. We have a methodology where we kick that out and run it on a host processor and then bring the work back. but it's unpleasant and we wish there were a solution for that. We thank you for bringing that up. That's just, you know, one of the problems were the early TensorFlow and even early PyTorch didn't really think hard about compilation. And the people who love Python love it in fact because it abstracts so nicely away from anything hardware. And that's exactly why it's so difficult to compile. And whereas sort of C or lower-level languages are more straightforward to compile because they force the author to contemplate data structures and where your memory is and these things are really important. So a trade-off was made in sort of favor of more users writing more code on Python and PyTorch and TensorFlow, which are, Pythonic, but they are by nature and by, because of the fact that they didn't really think hard early on about lowering, how you'd lower it, how you would sort of deliver it to an intermediate representation that we could all agree on. I mean, MLIR is a good step forward and, you know, Chris Latimer's work there is really, is really important to the industry. and there's others behind that, but allowing us to lower a framework and keep all the information that's important, right? That's what you want. And sometimes it's hard. I mean, compilers are hard. You were smart not to build one. It's not an easy thing.

No, it's a very impressive thing. I mean, I, you know, it's funny from where I sit, I, I think I really see the trade-off, right? Because, you know, I think, I think TensorFlow is probably a little bit more designed by people thinking about the hardware maybe than, than PyTorch. Because, you know, I feel like PyTorch kind of made it simple for an idiot like me to just like put like printouts in the code. And that's probably like, you know, part of why like I think PyTorch, you know, got so much, um, you know, popularity. But it certainly, you know, came at a and a cost. But I'm kind of curious, like, when I look at, you know, our own, like, user data and, like, you know, I think you have more advanced users than us, but we have pretty advanced users. I see, like, all kinds of, you know, really simple-to-fix issues with the hardware they're using that, like, even I could fix. Like, I see lots of our users not even saturating, like, the simple, like, you know, expensive, you know, NVIDIA resources, you know, that they have. And I guess by the time someone comes to you and buys your chips, maybe they're like experts at like doing it, but I would imagine that people probably still have trouble really getting like all the performance or even like just using like the entire chip that you offer.

Well, Lucas, I think a couple things. I think one of the things that big neural networks brought was they added a dimension of distributed computing to AI, right? When you could fit on a single processor, there wasn't a distributed compute element. But today, if you want to take, I don't know, a 20 billion parameter network and put it on 256 GPUs, you are going to spend months with Horovod, with other tools, thinking about how to distribute work. And measuring latencies and I mean, it's painful. We take away all of that. Whether you run on 1 machine or 16 machines or more or less, you do no distributed computing work. And so one of the things our compiler does is it allocates work onto the wafer. And so that was something we automated and as a result have very high utilization. And so that's something, you know, our compile stack has elements that look like EDA tools, it looks like CAD tool, right? We're gonna place work awfully onto the wafer layer by layer so that it achieves very high utilization and you don't have to think about it. Interesting. But the complexity involved in running a large, network on a cluster of GPUs is really significant. I mean, you're gonna begin doing some data parallel stuff and then you're gonna run out of that. Then you're gonna go to, you know, model, model parallel, tensor model parallel. And then maybe you're gonna go to pipeline model parallel. And so now you've got a 3-dimensional problem of parallelization and it is, it's, What's hard about doing big networks? It's not the design of the networks, it's the distributed computing that is mind-numbingly difficult. And that's something that we've been able to take to zero. We've eliminated it completely.

I guess, you know, I feel like in this last year we've seen a number of kind of ML-focused chip companies go out of business. What are your thoughts on like, you know, like what—

Thank you for bringing up good times, Lucas.

Well, it's a little funny, right? Because, you know, it's— we're like in the middle of like an ML boom. And, you know, you look at like— I think everyone sort of marvels at the, you know, the hold that NVIDIA has on the market. It also seems incredibly challenging to me to make chips, right? Because you sort of have to predict what the market's going to do. Like years in advance, but I feel like everyone making ML chips at least got something really right, which is that there's going to be like incredible demand for, um, compute. So I guess I'm like teeing up a real softball here for you, but like, you know, what do you feel like you, you got right that other people were, were missing? Because, you know, other people had that same insight as you in the sense of like ML being important.

They did. I think there have been some real challenges, and I, I think there, there are two reasons that some chip companies have have really struggled. The first is what they built wasn't sufficiently better. And I think as a startup, being a little bit better or a little bit cheaper is a terrible place to be, especially for hardware, because Nvidia pays less to have their chips made, right? They're one of the largest buyers of silicon in the world. They have huge partnerships with TSMC and with Samsung. So They're paying less for your per square millimeter of silicon. They get access to new geometries, new steps in fabrication first. And you've done something that is a little better than a GPU and it's mostly a GPU and you get to market and it's a little better and they just cut their price a little bit and everyone scratches their head and says, what, why? Move for a little bit of gain. And that's one of the reasons some of the companies have struggled is they're just not better enough. And we chose a very different approach and the approach had more risk in engineering. Every company that had ever tried to build a chip this big had failed, including Gene Elmdahl's company, right? A company called Trilogy. But we knew that if we built it, we were profoundly differentiated. right? And it's a real question for the entrepreneurs in your audience, where do you want to fall, right? You want to do a project that's a little easier, but that's a little less differentiated, or do you want to do a project that's got much more technical risk? But if you can get through the technical risk, you have a product that is profoundly differentiated. And so I think those were some of the things that we thought about and we have a clear opinion. We believe we want the risk. We talk about it in the building. We want our engineers to hold the invention risk and we want them to do something extraordinary. We, we're not afraid of failing. We're afraid of succeeding at ordinary stuff, at the mediocre. And so we didn't want to build a slightly better GPU. We wanted to build something different that crushed AI work. And so that's what sort of fires us up. And so we chose a much harder strategy. We chose an approach that required us to raise more money. An approach that, uh, had long periods of time when we were unsure if we could solve the problem. But when we did solve it, we had a, a solution that was clearly differentiated and empirically better.

And I guess you've been doing this long enough that, like, you know, chip manufacturing has improved, like, ML has changed a lot. Like, how many, how many versions of, of your hardware have you have you actually like created?

We began with a test wafer and that allowed us to test some of our ideas. And from there we did our first chip and that was at 16 nanometers. And then we did a 7 nanometer version and that's what we're selling. You know, you gotta run down next Wi-Fi and 3G and that's sort of the path. So, you know, we're pretty good at making chips. So, I mean, it's something we're pretty good at.

Is it? I mean, how much, like, you know, one thing I kind of marvel at with NVIDIA, I'm sorry to bring up a competitor so many times, but one thing that always amazes me is like, you know, I feel like new versions of CUDA come out and I'm just like baffled by the performance gains that they're able to get on the same hardware. And it would make me think that if I was in your shoes, I would need like a, huge— it'd be a huge undertaking, not just in the chip, but in the— not— and I guess not even just in the compiler, but I don't even know what layer you call it, CUDA, and then the sort of like, you know, kind of taking the kernel—

kernels. Yeah, the microcode.

Yeah, I mean, so I mean, how do you think about like even allocating resources among those, those undertakings?

Um, well, first, you know, NVIDIA has most of the market share, so I, I think it's perfectly reasonable we, we talk about them. And it's true that When they bring out new hardware, it takes months and sometimes years for them to drive up the utilization as they write new microcode. That's the software, that's the machine language that runs right on the actual hardware. I'm a professional David in the wars with Goliath in the chip industry. I mean, we, this isn't unique to them. This is true. And you have to know that going in. Right? That they're going to come out, they're going to do predatory pre-announcements, they're going to BS and say their chip got 7 times faster when, you know, if you don't get 7 times more I/O, it'll never get 7 times faster. And they're going to do a whole bunch of stuff and they're going to throw stones and they're going to, I mean, and underneath all the marketing BS, they're actually going to make their product better step after step if they're a well-executing company. And NVIDIA is a well-executing company. And so you have to know that going in. And so you have to have an architecture that's much better. You have to be ready for it. That's nothing you described is a surprise. It's the way the computer industry is run from processing chips for 40 years. Is that the hardware is the engine and the software's the brains. And as you get better at using the engine, the brains can run faster. And over time the same engine performs better because you've been able to wring out all the optimizations. And so that's known. You know, we knew what Hopper would be a year and a half ago, a chip that in most workloads is 1.5 to 2 times faster than the A100. We knew what they would say, oh, it's 7 times faster. We knew everybody who, knows about chips would roll their eyes and yawn. And the people who use it would get between 1.5 and 2 times faster unless they were memory constrained, in which they get zero. And so I think these are the battlefield, the landscape that one competes on and one knows about going in when one chooses to build processors.

I guess like one extra risk with, um, what, you know, you're doing is you're walking into kind of like a sort of like a nascent market. Like, I would imagine building sort of custom hardware for like an existing application to be kind of quite clear what the, what the market size is and what you could get if you really want it. Did you, like, how did you think about the market? Like, did you look at like inference and training separately?

Were there kind of other ways that you sort of like segmented exactly what you were You know, there are two ways to view the first part of your question. One is to say that it's harder when the market is emerging, when there's a dislocation. Even the customers don't know exactly what they're going to be doing in 6 or 9 months. And if instead you went into a mature market, nobody's changed in a long time, you know exactly what to build. Historically, mature markets favor incumbents. All right? They also know exactly what to build and they make fewer mistakes in mature markets. The dislocation created by AI in compute, the new challenges that have left AMD with zero share and Intel with zero share and gave us the opportunity, those are challenging for NVIDIA too. And in some sense, they level the playing field. It's never really level. It's always in their favor. But they, they give, historically speaking, that sort of dislocation creates opportunities for new vendors. And so we look for them. We look for the rise of new types of work. In the mid-'90s, there was the rise of a new type of work, and that was being able to do in hardware a memory lookup, an address lookup. And because we could do that in hardware, we began building switches and routers in hardware, and it created sort of what we think of today as data networking. In sort of the mid-aughts, there was a change in work where people for the first time in general compute said, what if we gave up performance in favor of battery life? And that produced new company. I mean, that, that's the cell phone. That's our, Right? How come Intel and AMD, and they knew how to make processors, why did they have zero shares? Because the landscape changed and the dislocation was what they used to value. Performance is no longer relevant and long battery life was really relevant, whereas all the other compute, you could always just plug it in. And so these dislocations are the places where great companies emerge, and that's where NVIDIA emerged. The initial dislocation that created a second chip off the processor that would do graphics and continue to leave graphics so important that it couldn't be sucked onto the processor. That dislocation created about 20 companies, of which the last two are NVIDIA and ATI, that was acquired by, uh, AMD.

Well, look, I totally agree. Like, I mean, you're really preaching to the choir that sort of like change favors the the small company. And that's, that's what I, you know, also look for as an entrepreneur, even within, you know, machine learning. I love it. Of course, you know, everyone like switches to something. I feel like we surely, we can move, you know, faster than, you know, some of the big cloud providers.

That's right. If we can't move faster than them, then we've got— then, as my old software coach used to say, Andrew, we're small but we're slow. That's a bad place to be.

But so I put myself in your shoes though, and it's like, I mean, what even is the the lead time, like if you kind of realize something's changing that you could take advantage of with like a new way of setting up your chip, like I guess I imagine that it's like years before that even. And also like I imagine that it's like hundreds of millions of dollars that you need to, you know, convince, you know, someone to kind of give you to even try it. So I would think that you would really need to have— you need to be very convincing about the market. Opportunity in a way that we don't need to be at Weights Biases?

Look, I think Weights Biases and other software companies, that the dynamic is very different in software, is vastly more modifiable. I think one of the fundamental challenges in hardware architecture, given and knowing that your chip will be delivered 2.5 years down the road if you're lucky, and that it's often unknowable what customers will do is deciding— Let's do it a bit more. What part of it is general and programmable and which part is fixed? And getting that right is the essence, one of the essences of being a good computer architect. It is, we, when we built this architecture, we had never heard of, nor had it been invented, had a transformer been invented. And yet we're the fastest in the industry at it by more than an order of magnitude. And that's because what we got right was not that there would be a transformer, but that it would rely on sparse linear algebra as an underpinning, and that we could build up from sparse linear algebra to solve what the AI community calls a transformer. And that is a fundamental part of computer architecture. Is, you know, had a different type of mask, right? Had everybody gone to single-shot learning based on a Bayesian approach using something else, we would've had a dead wrong architecture. But what happened was everybody stayed within a foundation that we predicted correctly, and that's that these problems could be broken down into linear algebraic elements, and in particular, sparse linear algebraic elements. We, we've been on that for a number of reasons, including there wouldn't really be a compute platform if they went a different direction, because we knew the GPU also required matrix-by-matrix multiplication. And so we used our knowledge of the landscape, our understanding of the history of compute, what was available out there to make some bets that we would be able to resolve a class of problems with this basis, this foundation. That's what you have to do.

And then, I mean, it's no question, but like, how do you know how many of these things to build? Like, how do you manage that?

Yeah, I dream of having zero cost of goods and no inventory challenges, Lucas, and yet I do my fifth hardware call. It'd be so nice to, oh look, I just, just send them one. Oh look, another customer on the SATs. How nice is that? You have to enjoy the challenge. You have to enjoy the complexity. We have hundreds of components. We have vendors around the world. You have to get your forecast right. You have to, have a reasonable estimate early on. You have to, you can build the largest and fastest chip on Earth. You can get your forecast right for the unit and a single power supply vendor can cause you not to be able to ship for months and months at end. I mean, how cool is that? That you have to, you gotta get that right. And that's why historically, hardware companies are worth a ton is because they are really challenging to get right.

Are there parts like outside of the chip that are big technical problems? Like I would imagine like the just the, you know, keeping the chip cool is an issue. Like, is powering an issue like does it even fit in like a normal data center?

Yeah, like rack. It's in a normal data center rack. So, so you're, so your audience knows our solutions. 15 rack units. So that's what, 26 inches, 26.5 inches tall. It fits in a standard 19-inch rack. It's about a meter deep. It comes in two flavors. It can be air-cooled or it can be cooled with water and it can go in a standard data center. But yeah, the delivering power to and the cooling It is, uh, uh, was a real challenge. I mean, it's a challenge for, for everybody. I mean, NVIDIA wanted on an, on a DGX, they wanted a cool front bezel. Only the one they made, you can't run the machine with it because it blocks airflow. And so that gold bezel is only for photographs. Uh, and he's like, wait, that's what I said. We had to invent some interesting techniques to deliver power to a chip this big, how to cool it.

Is it a liquid-cooled one so I can like put it under my desk and it's not loud? Is that, I mean, why do you even have a liquid-cooled version?

All of our versions use water to move heat off the wafer. In the air-cooled version, we have in the machine something that looks like a radiator that they don't like it when I say this, but it's, 110-year-old technology. Basically, it spreads the surface area of the water and then the fans blow cold air across it. So the exhaust is warm air. In the water-cooled version, we use water to move heat off the wafer and the water is then cooled with facility water in a water-to-water exchanger. And so for those facilities that have water, you run their copper pipes right next to your copper pipes. And so the the exhaust there is warm water.

What, does a typical data center have water to—

More and more of them do, yeah. Wow. More and more of them do. Yeah, it turns out water is a vastly more efficient way to cool than air. Cool. Yeah, so I didn't, I mean, the number of hard problems we had to solve, Lukas, in this project was extraordinary. And, It's been an amazing journey. Uh, you know, we, we're sort of fearless engineers. We, we think that, that really smart guys, uh, and smart people can confront a problem that they haven't confronted before using good engineering discipline and methodology and learning from mistakes and get better and better and better. And can do so without ego, can read the papers that have been written, can talk to people at events in the industry and learn and can produce exceptional results in multiple elements. You know, we see no reason why that can't be. And so we— that's why we attack these problems.

What did you What do you feel like you got wrong back in like 2016, 2017? Like, it seems amazing, you know, like what you got right. But if you could send a message like back in time, what would be the top bullet points?

You know, when we— by 2017, everybody was talking about ResNet-50 and Envision and we spent way too much time there. The GPUs for little networks were okay at it. We made a bet to go big on NLP. That was a great bet. If I'd have done it a year earlier, it would've paid way better. One of my board members said, you need to build a facility in Toronto. And I was like, that dude, man, he's always telling me to do I don't know how to do. And a year later I started building a group in Toronto. We have nearly 100 people and I wish I'd started a year earlier. I wish Toronto is an exceptional place for entrepreneurs to build out organizations. The University of Toronto has exceptional people. Waterloo has exceptional people. The engineering culture is first-rate. I should have listened. Then I hesitated because I didn't know and hadn't done it before. You know, the number of mistakes, and you know this, when you make 100 decisions a day as a CEO, the number of mistakes you can accumulate in a week is large, right? We, you know, you get 4 or 5 things wrong a day and you're doing great. At the end of the week, it's like, I made 20, how many bad decisions this week? I mean, how many mistakes did I make?

Well, I think especially in your case though, you know, the feedback cycles are a lot slower and the decisions seem harder. So I'm just— and, and like, when I think about like what I was thinking in 2016, I mean, that was a long time ago for ML. Like, I mean, TensorFlow had just launched, I think.

TensorFlow had just launched. I mean, the things we were thinking about, you know, Caffe2— should we support Caffe2? Should we bet on, uh, you know, there was even something, some other framework, uh, that I've forgotten about Yeah, I mean, this community has moved unbelievably quickly. And yeah, I mean, it's extraordinary.

And I guess like, what are you seeing now? Like, what are you like trying to support given that it's like 2 or 3 years before? Are you working on like a new and more version and like?

You always have to be. Lucas. What? You never stop. You're on the treadmill. You commit to that treadmill. You are on it.

So I don't know if this is like a top, top secret, but I'm just curious, like, what kinds of stuff, uh, like, what, what are like the new requirements that you like, you know, want to support in, in 2, 3 years?

You're always looking at what your customers' workloads are. You're engaged in conversations with them about what they want to do. You're looking at things that surprised you. You know, we bet big on, on NLP. Stable Diffusion and DALL·E, those were more surprising, sort of this interaction between vision models and large NLP networks. The models that are going text to video, and you're trying to unpack them to understand what are they doing the same and what are they doing different. And what part of your architecture that is fixed maybe ought to be flexible and what part of your architecture that you thought needed to be flexible, everybody's agreed on that and we can now be fixed. And so, you know, we'll give you an example from a competitor. In the A100, NVIDIA included circuitry for a 3x3 convolution. Because everybody was talking about sort of ResNets. Now, when you run Transformers, that ensures your utilization is abysmal across those circuits. There's an example of they went to fix and they got it wrong. And I think we try and, and understand, make bets, not at the application level, but at their underpinnings. Also, you have this list of things that you wish you'd have done last time but couldn't get in in time. You've got issues that had to be resolved, that you had to work around, and that's the nature of chip building.

Do you feel concerned that, or I guess this is like an opportunity, like I talk to some of our bigger customers and it seems like they're really essentially starting to build their own data centers around, you know, A100s. Like, would you deliver, do you typically deliver like, you know, like a whole bunch of your chips like networked together and— We do.

Right, more and more we're delivering clusters with millions and millions of cores. Yeah, I think that's a good sign for us. You know, I think, when is it really hard for a startup to hardware company when you've gotta go to many different enterprises and they want to buy one and you know, you need a sales guy in North Carolina and you need a sales guy in upstate New York and sort of the enterprise sales model, that's not really what we're seeing. We're seeing those who are in AI buying big and they're looking to deploy thousands or tens of thousands of GPUs. And if you can show them competitive advantage, you know, balls out there, you got to go win it.

And I guess where are you seeing the biggest pull from the market? This would be a useful thing for, for me to know as well, because I see it with someone spending as much as they need to spend on your, your stuff. They should probably spend a little bit on monitoring it.

And we have, we have lots of customers in common. You know that. And you guys have done well in the big pharma spaces and with some oil and gas guys that we have. Have in common. I think that the surprise for me of late has been this rocket ship that was Jasper and Stability and Adept. And I mean, this, you can keep naming them, Cohere and Anthropic, and the valuations, the businesses. I mean, you know, we announced Jasper was a customer of ours and got to know Dave a little bit and his team over there, and they're doing unbelievably big numbers up from nothing. They may be one of the fastest growing software companies in memory. And I think that was all unforecast a year and a half ago, right? It was like OpenAI and everybody's looking there and these guys, many of 'em are gonna drop, you know, 9 figures in training next year. And so all of those are new and interesting opportunities like that. Nobody was talking about 9 months ago.

It's funny, I guess another thing, you know, I feel like almost everyone— I'm almost everyone— many, many, many people do this stuff in the, in the cloud. Do you feel like it's important that, that you kind of offer a way to rent your hardware?

We do. I think for the big models, it's really hard to do in the cloud because you need dedicated clusters, you need guaranteed latency between your nodes. I mean, the largest node you can buy from AWS is 8 GPUs. You know, if you want 160, you're gonna get, you know, 28-node instances. And those instances, you've got no guarantees on latencies between them. What a mess. Oh, am I seeing cloud? Yeah. And this is important. What we're seeing is those guys who want lots go to people and ask them to do dedicated clusters that are leased. I think for most people under 30 in our industry, you know, they don't spend a lot of time thinking about the hardware, right? Hardware isn't a thing like I think about it. It's something you put software on. And that's, and so we have to change and adapt with the times. And so if they want to rent instances, if they want to rent cluster time, if they want to, rent by the model. You know, we announced last week an approach where you can pay by the model. You wanna train GPT-J to chinchilla accuracy, you know, it's $45,000, this many tokens, bring your data, let's go. You want to train GPT-70 billion?

Great.

It's this many tokens, bring your data, we tell you how long it's gonna take and let's go. You know, that, that's a different way to, to consume. And so we, we do think that, that providing. It's not skin off our teeth. We can provide. If you wanna buy it, buy it. If you wanna rent it, rent it. If you wanna buy the hour, buy the hour. I mean, it's the same.

I bet that's not a small undertaking on the, just on the software side to kind of, you know, make it reliable and consumable and all that.

Yeah, I think once you've compiled PyTorch, man, making it cloud available is not the hard problem. Uh-oh. It's a messy problem. It's not a hard problem. And I think it takes some thinking through.

It's funny, you know, we've obviously partnered with some hardware companies and I think I never, I mean, I'm over 40, so I'm not so young, but I never really understood what like waterfall development was. Like I think Agile sort of in response to this thing that I actually never witnessed, you know, until we started like engaging with some hardware companies. And I was really, it was just kind of like shocking to sort of see their like engineering mindset, which on one hand I like really, really respect it, but I think it would be— it seems like it might be hard for a company that's like, you know, like so geared up to like do these things with these like long lead times where you really need to get it right to sort of like build these things where you're really like iterating daily with, with customers.

Yeah, you know, if chip design takes 2.5 years, week sprints are probably the wrong thing to be engaging with. And That's why there are very few individual engineering leaders who successfully lead both. You know, we have a leader in hardware and we have a leader in software because they have different cultures. And even in software, I mean, if you're, if you're solving algorithmic problems or you're a PhD in optimization theory, right, you also likely aren't doing sprints every few days. On the other hand, other parts of the software organization are. And so I think there's no one-size-fits-all. I think thoughtful engineering leadership relates or brings together a methodology with the characteristic of the problem.

Yeah, that makes sense.

You know, software, upper layer software is sprints. Chips are marathons. They're marathons. And the guys who do the They, but what they're extremely good at is their marathon pace. His, and you know, you watch those East African guys who were the world's best at it and they're doing what? They're doing 26 and a half miles at 4-something, 4:50.

Yeah. It feels like a sprint to me.

That pace is mind-boggling, right? The best in the world. That pace is profoundly different. The way they train is profoundly different from the people who run 100 or 200s. And I think thinking carefully about what is the characteristic of the problem, what are the norms, how are you going to organize your engineering leadership team, uh, how are you going to organize your, your, your processes ought to relate to the problem you're trying to solve.

Well, look, we always end with two questions, and I, I think we might need to slightly reinterpret them to make sense for me, but I'd love to get your vantage point on this. Um, we usually ask, um, you know, what's an under-investigated part of machine learning or what's something that, you know, kind of warrants more study. I'm wondering if from your perspective, from your vantage point, you have a perspective on that.

So, Lucas, when I was working on my PhD, I would spend all this time in class working on the mathematics and statistics and all this time when I was doing research cleaning data, right?

I mean, yeah, I, I I know.

The gap between what you studied and what you did was so unbelievably large. I think an underappreciated area is the data pipeline, the managing of these interesting datasets. And I think sometimes it's kicked off by IT groups. I think the having a strategic vision around what data's gonna be collected. I mean, if you collected all, it has negative value too. You got so much you can't work through. Really, this area that's upstream of us is step or two, which is the strategy, the management, the cleaning, the storing, the data husbandry. I think is profoundly underdiscussed. The tools are nascent. Lot of opportunity there.

I totally agree. Um, all right, and, and final question is, um, when you look at the full life cycle of an ML model, from, from someone kind of thinking of it to, to building it, but then to also like in production, like doing something useful for someone, where do you see the surprising bottlenecks, or what's kind of the hardest part of that, that full journey?

That's really good question. I think when that model starts being deployed, and I think you just saw this with the new, with OpenAI's chatbot or with Meta's, I think data scientists sometimes, and AI practitioners approach development of a model differently than we would in software. I mean, I mean, thoughtful QA matrices, right? With an understanding of what has been tested and what hasn't been tested. Those are really hard. And so, you got your model, you've tried it, you did the distributed computing, painful, you've tossed your hair out, it meets a ton of data at first, and you're not sure how long that debug process is gonna take. And you know, when I was, a young engineering leader and I'd ask my guys, when the hell, when are we going to finish debugging? It's a head in plumbing the depths. You don't know. And he would make me so uncomfortable. I mean, there'd be physical uncomfort when you're peeling the onion and there's just more onion. You keep peeling it. All you can see is onion and onion and onion. I think that stage takes people vastly longer. Than they anticipate and could even prove, could benefit from some methodological improvements as well.

Yeah, I totally hear that as well. Yeah, we hear that.

Yeah, I think that that's what we see anyway.

No, that makes sense.

And it's a frustrating time. You know, we always tell our guys that last 10% isn't 10%, right? It, that last bet turns out not to be only 10%. It's a lot longer because you're not actually at the 90% mark when you think you are. You're at the 65 or 70% mark.

Well, I feel like in software that's bad, but in ML you may never get there. Like, it's like, it's not even like clear that, you know, what's actually possible.

So that's, you know, my, I mean, we've been studying things we don't know if there are answers to, studying theory, we've been doing science for years. I mean, there is a research methodology that's put to bear even when you're not sure there is an answer. And so I think that the community's learning and improving, but I think that there are joint pressures. There's a drive to get your cool new model out and get that big publication and have your Stable Diffusion moment where everybody goes, Whoa, like that, where a whole industry takes a gas and said, holy cow, and you can run it on your laptop, right? And you push that out and you want that so badly. But these are complicated things. It's gonna take a while before we roll them out so that we know exactly that they're fully QA'd and ready for the world.

All right, one final curveball question. I'm just curious if you have a thought on this. I feel like a lot of people have been noticing that the M1, the MacBook chips, like, are pretty fast for ML workloads. Do you feel like Apple might enter this market in a bigger way? Does that— is that something you think about?

Okay. Apple's the greatest company in the last 50 years, right? I mean, if you think of market cap and you think of sort of what— where they are in the economy, You know, that we moved from, you know, would you rather have your keys or your phone?

Right? I'd rather have my phone.

I mean, my phone's got a credit card in there. I can get wherever I want with Uber. I mean, they're better than your wallet.

Yeah.

They passed your wallet, they passed your keys, they passed, I mean, think of the things that they're better than. You know, they have tried on several occasions to do B2B and B enterprise. And it's been harder for them. That's a DNA shift for them. Yeah, that makes sense. And my guess is if they did it, they'd do it for internal use first rather than third-party service. But the things that they do know, they were among the first to realize it really, in your laptop, you know, faster processors, good, but better resolution monitor, it's really good, right? And so, uh, they have done a lot of thinking about joint optimization of hardware and software, in particular in display. If they wanted to, I'm sure they had the skills and expertise to build a part that could, could run ML well too.

Well, thanks so much for your time. This is really fun. If you're enjoying these interviews and you want to learn more, please click on the link to the show notes in the description where you can find links to all the papers that are mentioned, supplemental material, and a transcription that we work really hard to produce. So check it out.