No Priors Ep. 31 | With Cerebras CEO Andrew Feldman

The shortage of compute for AI, also known as the GPU crunch, is increasingly impacting AI companies big and small. It is delaying training runs and launches for multiple players in the generative AI world. One company coming to the rescue is Cerebra Systems, which is developing the largest computer chip and one of the fastest purpose-built AI processors ever. This week, Sarah and I are joined by Andrew Feldman, CEO of Cerebras Systems. Andrew is one of the few entrepreneurial veterans in the semiconductor world. He previously started C-Micro, a pioneer of energy-efficient, high-bandwidth microservers. C-Micro was acquired by AMD in 2012. Andrew, thank you so much for joining us today.

Elon, Sarah, thank you so much for having me.

I, I think you all recently announced that Cerebras has closed a $100 million deal with G42 to develop one of the largest AI supercomputers in the world.

We did announce a strategic partnership with a group called G42, and we announced that we were building 9 supercomputers. Each supercomputer would be 4 exaflops of AI compute, so in total 36 exaflops of AI compute. That was extraordinarily exciting when you encounter a partner that shares your vision and wants to build with you, and, uh, you get to start building the biggest computer on Earth. I mean, that, that, it doesn't get better than that.

And I think in general, you all were very forward-thinking and early to identifying AI as a really important market for custom semiconductors. Could you tell us a little bit more about your thinking early on as you started Cerebras and why you focused on AI many years ago?

In, in late 2015, 5 of us started meeting regularly. I think we were meeting in, in Sarah's office actually at that point. All of us had worked together at a previous company, and we began working on ideas. And one day our CTO, Gary, leaned back and he said, "Why would a machine built for pushing pixels to a monitor be ideal for AI?" Then he said, "Wouldn't it be serendipitous if 20 years of optimizing a part for one job left it really well suited for another?" That got us excited. We began looking at GPUs, looking at AI work, and by early 2016, we decided that we could build a better part for this work. And our strategy was not to build a little bit better, but it was to try and do something vastly better. And we went to a technology that had never worked before called wafer scale, and we build a chip that's sort of the size of a dinner plate, whereas most chips are the size of a postage stamp. And we, we did that because we knew that this workload would go big and that the problems of memory bandwidth and problems of breaking up work and spreading it over lots of little machines would be daunting. This is my 5th startup. This is the first time I was wrong on the market size on the underside. I had no idea it was going to be this big. And I think very few people saw, even those of us who were in it, how big this was going to be. And so, early 2016, we went out, we did 8 pitches, we got 8 term sheets, we raised money, and we're building multi-exaflop AI supercomputers for customers around the world.

It's really incredible thinking about the foresight. I actually went back and looked at my notes from like our end of 2015 meeting. And if I'm sure you remember very well, but for our listeners, uh, Andrew Feldman had a slide in there of how the top 7 problems for deep learning were all long training times. And like the whole industry was bottlenecked on compute. And this is even before the scale-up of transformers had happened. It's kind of wild how correct you were, at least on, you know, the depth of this problem.

I think what you need to do is save that snippet and send it to my wife. We got that right. I think also we got the fundamental architecture right in a large sense. We laid down the architecture before transformers exist and we're the fastest at transformers by a lot. So when you break up a problem and it encounters things it has never seen before and it's still really good at them, that's a sign in hardware architecture that you really got the architecture right.

What are some of the benchmarks that you use in order to assess the performance of your chip versus others? And how have you all performed relative to GPUs and other sort of more standard?

We are not big believers in canned benchmarks. When I was at AMD, we had a team of 30 whose job it was to game benchmarks. When our CTO was at Sun, they had a team of 70 whose job it was to game benchmarks. How long it takes your customer to train a model is the answer. And usually a collection of models. If you're trying to do a 13 billion parameter model, medium size, right? Almost always you begin at 100 million and you do 100 million, you do a series of sweeps, right? You're trying to do your hyperparameter tuning. So you do 100 to 300, 600, maybe a billion, and you're trying to find the right optimization points. And I think how you do on all of those is the answer. How long it takes you to move from customer model to distributed compute to finished run. And if you have to spend months doing distributed compute, doing tensor model parallel distributed compute— I mean, if you look at the back of some of these papers, they're crediting 20 or 30 people, sometimes more, who helped on the distributed compute. And if, if you don't need to do that, like you don't with our our equipment because we run strictly data parallel. Um, that's all time that you get back as a customer, right? That's all time that you get to train models and do interesting work rather than try and think about how to break up a large matrix multiply and distribute it over a collection of GPUs.

Can you actually talk a little bit about the architecture of your chip relative to others?

All right, it's a data flow architecture. It's comprised of about 850,000 identical tiles. Each tile is a processor and memory. And so, uh, it's a fully distributed memory machine. And so you have huge amounts of memory bandwidth because the memory is speaking to a, a processor that's one clock cycle away. So By data flow, it means that the cores wait until they get a token or a flit that arrives. It tells them what to do, do work on this and where to send it, and then it sends it forward. And this is a particularly good architecture for the type of work AI is. AI is a flow, right? It's a learning system. It's interesting, it was developed in the '80s at MIT by a guy named Arvind and Ben they didn't have a good workload for it, so it just sat there, right? Nobody— and then we got a good workload for it, and it's phenomenal for this. And that's a really interesting part of computer architecture, where a really cool architecture can have no interesting workloads for it, and then you get a good workload, and now it's ready. Now it's good at this particular problem. We keep a huge amount of SRAM on the wafer, all right? And so there are no memory bandwidth problems ever. That also allows us to harvest sparsity, which is something that others really struggle with. The cluster we, we build keeps the, the parameters off-chip in a parameter store, and it streams them in. And the result of, of this architecture is that we run strictly data parallel, which means even in a 64-node cluster, you run the exact same configuration on each machine. Each machine sees the exact same weights, but they see a different portion of the dataset, and it scales linearly, which is unheard of in compute.

Can you, for a broader audience, describe like how this addresses some of the challenges of using traditional GPUs for large-scale machine learning?

Sure. One of the real challenges of a traditional GPU is they fix the amount of memory on on the interposer, right? So you can buy a GPU with either 40 or 80 gig. What if you want 120? What if you want 20? What if you want a bigger parameter store, right? You gotta buy more GPUs even though you don't want those. And so what this strategy does is it allows you to support extraordinarily large models by disaggregating the memory from the compute. And in all GPUs, they're tightly coupled. And that means if you want more compute without memory, you still have to pay for the memory. If you want more memory without compute, you still have to pay for the compute. And that was an idea that came from supercomputing, uh, that we knew really well, that we could organize this so you could run an arbitrarily large— now, a trillion-parameter network on a single system. Now, it would be slow. That's a very big network. But you could run it on one system. You can identify layers. You, you can debug. You can do all this. And you can never do that on GPU, right? Because there's nowhere to store the parameters. And, and so by separating parameter storage from compute, we allowed you to mix the various ratios at your will rather than at the GPU vendor's will.

So I guess in terms of the relationship that you have with G42, there's a variety of different applications that you've looked at across healthcare, across a variety of different verticals. What do you view as some of the first use cases or best applications for your architecture?

So the, the first thing that they allowed is a, a model called BTLM, which is the number one 3 billion parameter model on Hugging Face right now with more than a billion downloads. We have Fortune 100 companies running on it right now. We have them doing their own internal work on it. On Wednesday, we're announcing that we're open sourcing, uh, with G42's group called Inception and MBZUAI University, the largest Arabic LLM. So we're putting that into the open source community. That's a collaborative effort with our strategic partners there. When you have a lot of compute, the mind boggles at all the cool stuff you can do.

Why did you guys get into the training and open source model game?

We had so much compute. And that it, it was a way we could prove to the world how easy it was to train on us. We felt it was evidence that we could build and our systems could train the biggest and fastest models, the most accurate models in the space. In March, we put 7 GPT models in the open source community. Everybody else was putting 1. Why? Because it's really hard to redistribute work across a GPU cluster. For us, it's one keystroke. So we put 7. People are coming to us with extraordinary ideas. We had a customer who came and they said, look, we'd like you to design for us a model at 3 billion that we could prune and quantize such that it could be served off a cell phone. Really cool.

What are the biggest milestones that you're thinking about going forward?

There's been a lot of back and forth. Everybody thought we'd keep getting bigger. You know, north of $175 billion is a giant network. It's expensive to train. And maybe even more important, it's really expensive to serve, right? 17.5 times more expensive to serve than a 10 billion parameter model. And so I think a lot of the models in production are 13 billion and smaller. As an effort to keep the cost of production down. We are training right now a whole collection in the 30 to 175 category. I think there's going to be a spike in multilingual models, right? Stability just put out one in Japanese. There was one in Spanish that came out. Inception, MBZUI, and Cerebras put out one in Arabic. I think these are underrepresented languages. And I think we have a challenge there in that most of the big datasets are giant internet scrapes, and most of the internet's in English. And so it's really hard to build big datasets in, in languages that aren't English. We don't have enough tokens. And so techniques to carefully think about how you can do that, I think, will be very much in demand. I think every nation wants to feel like its language and culture is captured nearly in an LLM.

That's interesting because a lot of the training also seems to capture social norms versus data. And one could imagine as well, as you think about different regional models or different language-specific models, people want their own set of cultural norms to be part of the model in some sense versus something that's imposed.

That's exactly right. Language captures experience. I mean, there's words that, that are different and have very unique— I mean, it's just like the New York experience includes Yiddish, right? It will be important that we find ways to capture these cultural experiences. And we do that in language. And if you overwhelm the minority language with English, all that gets sifted out.

I think the Yiddish LLM would just complain all the time.

That would be hilarious. That would be hilarious. Yiddish LLM. And it's like, What, your brothers are doctors?

Yeah, every response is, why aren't you a doctor yet?

Are you going to finish your PhD, Andrew Feldman?

Oh my God, no, don't worry, the Chinese LLM says that too.

Oh, it will.

That's right, that's exact same, same mothers.

I, I think it's like actually like kind of a deeper insight in the joke, which is there's cultural experience baked into both the actual tokens and pre-training curation and like what you're weighted toward, as well as in the LHF layer, right? Because whatever, you know, human annotators are saying like, this is what I want, somebody wants to kvetch with their chatbot too.

That's exactly right. One of the things we learned was that it's really important taking into account the language that has fewer tokens, especially, you know, Arabic, Hebrew, Hindi, their characters sometimes require 2 bits rather than 1 bit. And so the tokenization scheme, has to be really thoughtfully done or you further bias the model in favor of the English. I think, you know, you put your finger on exactly why our strategic partners are so excited by this is they want their culture and their language properly represented. They want it reflected. They want the nuances in their culture reflected in an LLM and not sort of sanitized or washed with, you know, 99% English tokens.

Can we zoom out and just like talk a little bit about what it's like to start a chip company? It's a very uncommon experience, especially, you know, in as far deep as you guys are now with the money you've raised. Like, what's the hardest part of bringing a new chip to market?

I, I think the structure of the chip market is, is challenging. I think one of the things that, that, that VCs love about SaaS about software companies is for not very much money, you can get a prototype and you can start seeing if customers like it. Chips are the opposite. We're going to spend 2, 2.5, sometimes 3 years and $50 or $60 million building our first part. And then we will learn if people like it or hate it. It has a huge gaping chasm between when you start and when you begin learning from from customers. And that's one of the things that software doesn't have. Now, it turns out that if you're building a SaaS company, you end up often using as much money. You get insight into what your customers like much, much earlier than in a chip company. And I think that's perhaps the biggest structural difference. The structure of engineering is different when you've got a 2 or 3-year project. You can't do 150 one-week sprints. The way you manage QA versus development is different. Often in chip companies, for each developer, you have 3 or 4 QA people, DV people. And that's because a bug can cost you $20 or $30 million. It's not like Alfred said in 4.1.1. You're going to have to re-spin the chip if you have a big bug. And so the commitment to tooling, to QA, to simulation is profoundly different in, in chip— in the chip world than when you're building, uh, software. Now, we still have to build a huge amount of software, and we're about 75% software. When you build a chip, the chip is the muscle and the software is the brain. You still need a huge amount of software. You need low-level software, software that runs right against the hardware. You need a compile stack that allows the, the user to write in PyTorch and just run as if, as if it were a GPU. And you end up with, uh, a lot of different types of engineers in a system company.

Andrew, you have such a unique expertise and depth in a question that everybody is thinking about right now. Can you give our listeners an overview of, um, the AI accelerator market right now? Like, who are the big buyers, and then how might they decide to do anything but NVIDIA?

NVIDIA's made hay when the sun shines, and and they've done an extraordinary job, and you have to take your hat off to them, right? I think they're now in a situation where they're extorting customers, they're extremely expensive, they're unable to ship. And that has, among other things, opened the door for many of us who, who have alternatives. Nobody likes being dependent on their vendor. You can ask the guys at Google and Facebook who were dependent on Intel for years how much they hated that. They dislike that intensely. And so I think there's a battle between people's sort of dislike of being dependent and the need just to keep running forward. I think large enterprises continue to be a good part of our business. I think often where you can provide a little consulting to help them accelerate their model deployment is something that, that provides an opportunity for for smaller companies to get in the door. And when you show them how, how much easier it is— we had a situation where they were trying to train on a GPU cluster and they were at 60 days and it wasn't converging, and we stood it up and 3.5 days later their model converged, and they're like, holy cow. We have an internal cloud so customers can just jump on your cloud, they can begin training right away. If they want to try before they buy, they can begin with a little training run and go from there. We have customers in large enterprises and generative AI startups. We have customers in the military and the intelligence communities. We have customers in the supercomputer market world. We sort of up and down organizations of different sizes.

How much specialization do you think there'll be in the semiconductor layer going forward? In other words, do you think there'll be increasingly different chipsets for inference versus training? For transformer-based architectures versus diffusion models, for multimodal versus not. I'm just sort of curious how you view all that.

The fundamental question in computer architecture is what do you make special and what do you make general? And the second fundamental question is what are you good at, what are you not good at? What do you choose not to be good at, right? And those things are really important. You know, if you put in circuitry for transformers, that has a cost. There's a penalty every time you do a convolutional network. And so these trade-offs aren't free. You put in circuitry for one thing, if you're not doing that thing, that circuitry just sits idle and you've wasted power and space. And so I think it is a TBD question. On whether we will continue to see more specialization at the chip level in training. We do not do it. We crack the problem in a different way. All of the problems are linear algebra and are predominantly sparse linear algebra, and so we focused on underneath, uh, the Transformer. Now, whether you'll have different silicon for training and for inference, I think you will.

So your view on this, Andrew, is less a strong point of view that there will be something beyond transformers that is very important in the near term, and more you've already solved it in a more general way.

That's right. The trick in architecture is to, to, to solve hard problems in a general way. So you don't have to rely on product management to sort of guess what, what's the next cool layer type. Right? If we go to a type of model that doesn't require very much data, if we go to single-shot learning, right, NVIDIA is totally out of luck, right? Um, us too, everybody. These models are designed for extremely data-intensive Transformer-based, but whether you're doing convolutional neural networks or Transformers or diffusion models, or certainly new things will be invented. I mean, we haven't run out of ideas. We haven't even run through the ideas that the guys put together in the '80s, right? We still have plenty of work to do to improve our model making. And as the industry matures, and we're still in the early innings, you might see more customization for a particular model type. But that's not my preferred approach.

Are you making bets internally on separating training and inference?

We make all sorts of bets, Sarah. We lose frequently, and we win sometimes. That's the real fun of building companies in a dynamic environment, right? I think inference, especially generative inference, right, is a very different problem than all other forms of inference, right? Basic classification is the exact same as the first pass in a training run. But when you're doing inference on generative AI, It's not, and you have to hold a ton more information at the inference point. And so I think there will be— we will discover, we will invent new and different ways to attack that problem. It's extremely expensive right now. I mean, people using H100s to do inference on a big model, I mean, that's half a million dollars.

You know, there is this ongoing crunch to supply for GPU and other sort of AI-centric compute. and that's causing different people to delay launching their companies or training their models. And those are often the same thing for different types of AI companies. What do you view as the main cause of this supply crunch, and when do you think it loosens up, or when do you think things loosen in the market?

The chip market has a profoundly inflexible supply chain. And so TSMC, their fab is a pyramid in modern society. It is the absolute peak of manufacturing capability. It's a $20 billion building in which nobody works in it, and ideas come in and out come chips that cost $100. I mean, these are unbelievably, unbelievably amazing things, but they can't turn on a dime. And so when somebody misses their forecast by a lot and they come back to the fab and say, I need twice as much or 3 times as much, Often that takes 6 or 8 months to sift through. And that's the problem that's coming in right now. I mean, it's not just that Wall Street missed what Nvidia would sell. Nvidia missed it. They missed the forecast. And then you can go to TSMC and you can ask them for wafers, but many of their wafers are already allocated to AMD. They're allocated to Qualcomm. They're allocated to other giant companies. Right? And so you struggle forward. And so in addition to taking a long time to get to, to, to first customer learning, there's a huge premium placed on the ability to forecast accurately in the chip business.

Given that we're going through this big discontinuity from a technology perspective, adoption perspective, et cetera, do you think we'll be accurate in forecasts going forward anytime soon, or do you think there's likely to be a crunch for the next couple years?

No, we're going to continue to get it wrong. We're going to continue to get it wrong for a while.

Do you think we're largely going to underestimate it?

Oh yeah.

So basically the market is accelerating rapidly, everybody's underestimating it, and, you know, fundamentally we all think that this is going to be many times bigger than anybody thinks 2, 3 years from now.

Yeah. And the dynamics of our supply chain are when you get it wrong and you order too much, you still have to take it. Right. You don't get sale. Guess what, guys? I really don't want those wafers anymore. And they're like, see this $10 billion worth of wafers? Yours.

Yeah, that happened during crypto, right? So I think Nvidia had a really big miss during crypto.

It happened during crypto. It happened to Broadcom. I mean, Arista today is at 52-week lead times on switches. Mm-hmm.

Given that you've been involved with training some incredibly performant open-source models, And you mentioned that as you scale up a model, inference cost starts to really kick in as you get to bigger and bigger models from a parameter perspective. What do you view as the limits to scaling these models? At what point do they get too big, or do you think we just keep scaling them until we have these hypermodels?

It's a really interesting optimization problem. They get big, they get harder to work with, it's harder to retrain. Even GPT-4 came out, it didn't have any insight beyond 2021. They had race. I mean, that was because it was so big, right? And, and so I think there are trade-offs. There are trade-offs between accuracy and size. There are trade-offs between size and cost to do inference. I mean, maybe I want a larger model for radiology files, right? Or for my doctors. Maybe I'll take a smaller model for my, for my chat or my customer service bot, right? We're going to have to think about these in, in terms of what they're delivering, what the cost of being wrong is, what the cost of serving is, and think about this as a business decision. And at first, everyone's just running, trying to say, I can make a bigger model, I can make a bigger model. And then the guys who are trying to run businesses, you're like, well, I can't afford to give that away. And so we're going to be down at $3 billion and $6 billion and $13 billion because that's— I can get pretty good, pretty darn good and not break the bank with free inference. And so I, I think that there are these trade-offs that are really challenging, and we're just beginning to grapple with now.

I guess OpenAI, related to that, recently announced a partnership with Scale around fine-tuning. How important do you think fine-tuning will be going forward?

I think fine-tuning will be really important. We're still just learning about the capacity of these models to hold information and to hold insight. And fine-tuning is an extremely exciting approach. I think we've seen again and again that, that, that human feedback makes these models vastly better. And that, that's not surprising, right? I mean, that's how you learn to dance. That's how you learn to do everything— soccer, gymnastics, ballet— and somebody saying, no, that's wrong. Nope, do it again. That correction is a mechanism for improvement. Somehow doesn't seem very surprising to me. I think we don't yet know how far fine-tuning is going to take us and who will benefit from a model trained from scratch versus continuous training versus taking a model off the shelf and adding to it their unique data.

You know, there is this point of view from some of the large labs that very few people will do training. And it will get quite concentrated. But I've heard you speak about how you think that there are really interesting datasets in different large enterprises. Tell us about that.

You know, at first everybody scraped the internet, but then everybody had that. And now the, the companies that have extraordinary datasets, Reuters, Bloomberg, GlaxoSmithKline, AbbVie, right? These companies that have years of research and exceptional data data are going to, I think, step into the fore. The AI has made progress to the point where people are looking around for exceptional datasets, unique datasets. The rise of data as an asset, I think it's the new gold. And I think that's something that isn't talked about enough and will be very much top of mind going forward.

Find us on Twitter. @nopriorspod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.