Launching the fastest AI inference solution with Cerebras Systems CEO Andrew Feldman

You're listening to Gradient Descent, a show about making machine learning work in the real world. And I'm your host, Lukas Biewald.

You know, what we've just announced is inference, and we are now the world's fastest at inference. We are at the most accurate level, and we're the cheapest.

Today I'm talking with Andrew Feldman, the CEO of Cerebras Systems. Cerebras is a super cool company that makes giant wafer-sized chips that work really well for ML workloads. It's a tough business and a tough company, and today they have a really exciting announcement that he came back on the podcast to share with us. I hope you enjoy this conversation. All right, Andrew, thanks for coming back for a second, um, interview.

It's good to be back, Lucas.

Good to have you. I was actually going over the notes from the last one, and I feel like I really got in all my electrical engineering questions, and I appreciate you bearing with me on that one. So I'm gonna, I'm gonna—

I know that you, you're an electrical engineer just hiding as a software engineer. I know that's exactly what's going on.

I want to be an electrical engineer, and I can't believe that your, your degree is in like political science or something. I don't know how you got so technical.

No, it's, uh, it was a mistake, uh, to study political science and not to do more engineering in school.

Well, it seems like you made up for that in your, uh, career, clearly.

Yeah. Yeah. I think, uh, 30 years of building chips and, uh, uh, they, they've given me an honorary degree. Just don't ask me to solve differential equations. That's exactly right.

Well, don't ask me that either. Um, But, uh, but to catch me up, so what's, what's happened in the last year? It seems like it's been a big year for you guys.

It's been a great year for us. You know, we've, we've built, uh, tens of exaflops of compute. Uh, we've stood up some of the largest AI training clusters in the world. Um, we have, uh, trained models that are in production that have solved fundamental problems and drug design. We have still the premier Arabic language, Arabic-English chat model, a big LLM. We've published papers in which we have furthered the state of the art in molecular simulation and seismic analysis. So it's been an extraordinary run for us.

That's fantastic. And I really want people to go back to the last interview for more technical details, but the highlight for me is always with you that you make these gigantic chips. But where, I guess, where, what are the, what's the strength that you've leaned on to do well in so many different domains? What are the axes of the chip market and like where, what's your like your sweet spot where like your chips really excel?

That's a really thoughtful question, Lucas. I think—

Real softball, Andrew Feldman.

No, no, no. I think people who tell you their stuff is good at everything are almost always BSing, right? I think that when you choose to do an architecture, you make some trade-offs. You're gonna be good at some things and less good at other things. On the training side, if you fit in one GPU, right? So that's a small model. You should probably use a GPU. It's a really good machine if you fit in it. But as the training work gets more complicated and say you want to do work on 100 GPUs or 1,000 GPUs or 10,000 GPUs, the actual work of taking your model and distributing it across this compute landscape is punishing. And that's where we excel. That's where the fact that we have such a large chip makes the distribution of work trivial. And once the work is distributed, the actual training is blazing fast. Now that's on the training side. And you know, what we've just announced is inference. And we are now the world's fastest at inference. We are at the, the most accurate level and we're the cheapest. And so you have this sort of trifecta, right? Fastest, top accuracy, and lowest price. And not by a little bit. right? Where if you compare us to what you can get from an NVIDIA H100 on Azure, we're over 20 times faster at inference.

And is this, is this for what, what models are we talking about here?

Let's just begin at Llama 3.1, 8B, and 70B. And so, you know, these are the, the two most popular models and we're, we're delivering speeds that are impossible to achieve with GPUs. So if you care about latency, right, how fast your, your answer is delivered. So if you've got an agentic model, if you've got, um, a concatenation of models where the output of, of one model becomes the input of the next, you care desperately about the amount of time it takes to generate the answer.

But those models actually aren't that big, right? So— No, they're not that big.

They're not that big, but they are inference. Generative AI inference is a very hard compute problem. And it's hard. We'll dive down into a little bit of the electrical engineering you love so much and that undoubtedly bores your listeners to tears. Each token, that is generated has to go through all the parameters, all the layers and all the parameters. And so to generate one token, you have to move all the parameters from memory onto the compute, generate a token. You then have to do that again for the next token and again for the next token. So if you are running even an 8 billion parameter model and the parameters are 16-bit, parameters, you're moving huge amounts of information, data, from your chip, from your memory, to your compute over this tiny little pipe, memory bandwidth. And that's the bottleneck. And so you're zipping this back and forth. And because we built a big chip and because all our memory is on the chip, we don't have that problem. And so it's blindingly fast. I think so for something like the small models you're describing at 8B, it's order sort of about 1,750 tokens per second right now, which is, you know, more than 20x faster than the hyperscalers on H100s.

And but, but aren't the parameters the same at every step? Like, why, why does the memory have to transfer the the value of the parameters back and forth?

Because you have to redo the analysis, right? You've got all your prompt and you add a token and then you need to redo it with the next token and the next token. Each time you go through this full cycle of moving memory, moving the parameters from the from the memory to the compute. And this process requires this sort of toggling back and forth. And remember, memory bandwidth is one of the weaknesses of the GPUs. And so this exact, uh, act of generative AI puts pressure on one of the weakest parts of the architecture.

Do you have the same accuracy and throughput trade-off with these models? Like, is it a whole curve that, that you can let people pick their, their point on?

Sure, we, we are running today at 16-bit at all stages, right? So that's the, the top level of accuracy.

Sorry, my bad, I meant throughput latency trade-off.

Oh, right. So the GPU allows you to add users, right? But to go slower for every user, right? Right. That's a trade-off you can make. Um, those who go really fast have a very small number of users. So, you know, some of the guys like Together or Fireworks are sort of leaders, or Octo, go really fast, but they can't stack many users on at a time. And we don't have that trade-off. We're blisteringly fast and we can stack many users simultaneously. I see. And so that's a very attractive component and that's what enables us to drive down the cost because we can be fast with many users simultaneously on the equipment.

So you're now running your own service that runs any model or specific models? How does that work?

Sure. So we, this is our inference cloud and today we're running Llama 3.1, 8B and 70B. And over the next weeks and months, this will expand to a full set of leading models on the service.

That's very cool. How long has this been in the works?

Oh, about 9 months. I think one of the most interesting things that we have seen this year is the movement of of sort of AI from hobby to production, right? And from cool thing to tinker with to something you can try and, and build business applications on top of. And I know you guys have seen that at Weights Biases and we've seen it across the board. What's interesting is that the amount of AI inference compute you need is a function of the number of users and how often they use it. And the size of the model. And so as more people use applications that embed AI in them, as they use them more often, the demand for inference compute goes through the roof. And I, you know, estimates are today it's about 40% of the compute, the generative AI market, and expected to continue to grow faster than the training market, though the training market's continuing to boom.

We've seen, we've obviously noticed an explosion of companies doing inference, and it does make sense that inference should be more of the compute load than training. Otherwise, these things seem just unbelievably wasteful. But it does seem like inference is kind of a challenging business because the comparison is so is so stark between throughput and accuracy on a given model. And I feel like we've seen this real boom and bust cycle with inference providers where one company puts out an amazing result and then a few weeks later another company puts out an amazing result. And it doesn't seem like there's a ton of lock-in when the API is literally like a text query. I'm curious, how do you even like, think about how many chips to assign to something like this? How do you, how do you forecast demand for a launch like this?

Well, first, your observation is right. It's that if, if, um, you know, if everybody's using the same hardware, um, then what you're doing is you're fighting for software optimizations on that hardware. And that's a tough battle. And that's why we went to a fundamentally— that used fundamentally different architecture, right? I mean, where all the GPUs are and TPUs are based on HBM, they're off-chip memory. They are— they all face the same constraints. We're a wafer-scale chip, we're an all-SRAM chip. And so we attack this problem and achieve solutions that, and performance that they can't achieve no matter how much money or GPUs or time or effort because we have 7,000 times more memory bandwidth. And memory bandwidth is a binding constraint in this workload. So, you know, you can't go faster than the speed of light. They can't get more packets. Between memory and compute than the memory bandwidth allows. And so that, that you got to attack it architecturally is the answer to your question. If you want meaningful and durable advantage, Lucas, you got to have a better architecture. That's number one. I think number two, you said, how do you prepare? You make an educated guess. I mean, I, I think people would like CEOs to always have the answer and to, to have a, an illusion that that much of this isn't, or that there are good equations to tell you how much to prepare for. You sit with your team, you talk to your customers, you look at what others have done, and then you make an educated guess. And that's how you start. And then you iterate quickly, right? You know, in many optimization problems, you begin with a seed and you iterate off the seed, and you begin with an educated guess. And you're ready to respond extremely quickly to when you get it wrong.

Okay, but I think it'll be hard to iterate quickly if the iteration requires manufacturing a ton more, um, ton more of your hardware.

Step one is don't have this be your first rodeo. That's number one is, you know, you have a team who's been delivering world-class chip for 30 years. So you have a plan. And I think you also have capacity in that you've allocated for training. So you have some fungibility.

Right.

And so I think you also have— you've notified your manufacturing partners long time in advance that we may be seeking to pull forward equipment. And so I think, you know, I made it sound like you sort of guess, but in the business of having to forecast things that are very hard to forecast, you try and manage all the unknowns, right? You work with your manufacturing partners to be able to pull forward equipment. You work with your customers to say, You know, I've got a new launch here. We'll get you on this service. If you need a huge amount more, here's a date you can have it. And you set that expectation in advance. But just about everybody gets it wrong at the start, right? I mean, there's no— the forecasting of unknowable things is pretty hard.

Totally. Totally.

I mean, it's the same in every business. We just sometimes we pretend that we're better at it than we are. And that there's some science beyond doing all the thoughtful things I described.

And where are these, where are your boxes going? They're kind of an unusual shape, aren't they? Like, do you have to build your own data centers or how do you do that?

Yeah, that might be a bridge too far. I've not undertaken physical construction of a data center. I like to stay at 5 nanometers or smaller, the things I build. Um, our, our, our chips, our wafer-scale chips fit into a chassis we design. The chassis fits in a standard, uh, data center rack. And so, you know, we have hundreds of machines here in, uh, Sunnyvale. We have machines in Stockton, California. Uh, we have machines being stood up as we speak, uh, in Dallas, Texas. We have contracted for power and space in Minneapolis, Minnesota. We have partners that have power and space and our equipment. So I think, you know, that's what you do. You rent power and space by the megawatt and build out a big cluster.

If your throughput is higher or your latency is lower, the same accuracy by an order of magnitude, why would you also price it lower than comparable solutions?

Oh, I think we intend to crush this market. I mean, we intend to go at this market with force and we think that the following obtains, you know, if you and I both remember when the internet was slow and when, uh, Netflix was, was mailing you DVDs, right? Um, the, the internet got fast and they started streaming. And the internet got really fast and then everybody started making little movies, right? That when you make infrastructure fast, you enable multi-billion dollar markets to sit up on top of it. And I think right now we're in the dial-up era of inference. It is slow. And when you're waiting for your response, you can almost hear that modem noise, right? That we used to get when the modems were hooking up. And I think our desire here is to make this market bigger and to show people that there are all sorts of things that aren't obtainable when inference is slow that are, are dazzlingly interesting. Here's a very simple one, for example. If you ask Llama 70B to do a math problem, Um, often it will get it wrong. If you ask instead a harder question, if you say, walk me through the logic you use to solve this problem, now it will take more tokens, it will walk you through, and it will frequently get it right. Now imagine you can do this, the latter task, in the time it used to take to do the former task. Yeah. Now you got a better answer. And so here's a very, very simple way in which speed turns into higher quality answers. Totally. Now, right? And many of us now are using our own input prompts iteratively, right? So we ask one of these models a question and then we use the response as we ask it to improve the response. right? Now, uh, you write a little, little bit of code for that. And now if you're 10 times faster, that happens in, you get 10 iterations to 1. If you're 20 times faster, so each time you're getting better quality results. And that's what we're trying to deliver to the developing community. And that, that's sort of the, the vision.

You know, we, we see a lot of customers that, um, they increasingly want to build complicated guardrails that involve calling LLMs in parallel. So I could see, I could see that really being, um, unlocking—

usually important.

Yeah, hugely important.

Um, you, you want to use them for guardrails. You, you have a security posture. Um, you want to do a check, uh, to manage hallucinations. Um, there are all sorts of, of simple benefits in the existing mode before you even go to breaking up the prompt into hundreds of, of mini prompts and kicking out into an agentic world to reconstruct, uh, a response.

Do you, um, I mean, it's an interesting space that I don't, you know, know a ton about. Like, I feel like you've taken this real posture of kind of like David versus Goliath with NVIDIA, and really, I mean, you've been one of the few people really kind of willing to take shots at Jensen, both to journalists and I saw it to his face at a conference we were both at. And do you feel like NVIDIA is really your target or do you worry about— and I think there's a whole bunch of companies with kind of different takes on chips that are finding the inference market lucrative and finding various axes to compete on.

Do you worry about those too, or is it really kind of NVIDIA as the As far as, you know, I, I have taken some, some shots at NVIDIA. They are the Goliath. I think also, uh, as I've told you, I think there are 3 great CEOs of the last decade in our industry. And there's Hock Tan at Broadcom, there's Lisa Su at AMD, and there's Jensen Huang. Right? What, what those companies have done is extraordinary. And if 10 years ago you had a choice, you'd want to be, uh, their investor. He could want to be in their stocks, all of them. And so, uh, because I'm going after, uh, and competing, uh, it doesn't mean I don't have enormous respect for, for each of them and think they're, they're, they're sort of achieved legendary status. Now that doesn't mean I, I don't think I've built a better product. We don't think we've built better product for this work. It just means that, um, you know, you, you, We are David against Goliath and they have approximately all the share. And so, you know, talking and fighting with other startups, I wish them well too, right? The challenge is to beat the H100 and then beat whatever's after that and whatever's after that. And I think spending time duking it out over crumbs isn't the right strategy for startups. I think the right strategy is to go where the market's big to go after the, uh, the, the biggest players, um, and do things that, uh, that they do better. And not a little bit better, but orders of magnitude better.

So, so you also— I guess, are you saying that AMD also feels like a target? Because I think from a, from an ML perspective, they kind of feel like underdogs. I think to most of the audience—

no, I, I think, I think, I think Look, they're going to have, you know, from what we see in the market, they're going to have a great year. They're going to sell a ton of their parts. But I think they are the underdog. They're just a bigger underdog than we are. Right. And when you're an underdog, size is not always an advantage.

Totally.

Right. You know, my old soccer coach used to say, Andrew, we're small, but we're slow. Right. And, um, the irony there is, is the hope is that what you get by being smaller is, is alacrity, speed, is, is ability to adjust quickly, is, um, ability to hire the, the absolute best people who just want to do engineering and don't want to deal with a collection of other things that come with, with being at a big company. But, um, look, we, we're a big customer of, of AMD. We use their EPYC processors and their Genoa processors in our training solutions. We have tremendous respect for the work they're doing. And sometimes you compete with your friends too.

Man, you're a little less spicy on my interview than some other interviews. I thought maybe I'm not as good of an interviewer.

Maybe you're seeing learning in place. I don't know. Maybe you're witnessing a learning algorithm happening here. I think, look, Lisa bought my last company, C-Micro, and I got a chance to work with her and she's extraordinary and we have the utmost respect for her and for Mark Papermaster and Victor and the team they've got over there. But I think when you're a professional, David, And that, that's what I am and that, that's what my career has been. You want to compete against the best, right? You want to get to the big leagues. You want to face the best fastball and the best curveball. And so thinking that, you know, those are the best CEOs of the business and how can we beat them? How can we, right? That seems to be what's fun about the business. Right? How can we deliver to customers using the advantages that we have? An ability to do extraordinary engineering, an ability to be fearless, to solve problems that others struggle with. And, you know, the recent announcements around NVIDIA's delays, I mean, these are problems we solved years ago.

What do you mean?

We attacked them years ago and solved them years ago. And, um, that gives just a, a sense of, um, sort of what you have to do to, to, to, to be a, a successful challenger.

Well, so sorry, I'm not totally up on, on, on the chip world.

You're not up on rumors on, on the chip space, uh, Lucas? Why not? I mean, come on.

I'm sorry, not also bad research. Yeah.

Don't you have a team of researchers, Lucas, who are preparing all this for you? Yeah, they had some packaging challenges that have caused some serious delays in their next generation parts. And the problems there have to do with, in our estimation, a set of problems around coefficient of thermal expansion and how you put chips on interposers. And these were things that we thought about and problems we saw, you know, half a decade ago. And so, uh, I, I think, uh, you know, that when you see your, uh, when you see the work you've done paid off, uh, uh, that, that's, uh, that's rewarding.

Um, well, okay, so I'll say from my perspective, one thing that I've noticed in the past month or so, you know, it's been sort of like, you know, rumors that maybe chip prices are coming down. And I have no idea, but I have had a lot more reach out than typical of random people trying to sell me H100s and things like that. Do you think there's any, like, truth to that? Or could there be, like, a contraction in in demand?

I think that brokers have a flood of parts, right? And they're the ones who are reaching out to software guys who don't buy hardware offering parts, right? That's not a thoughtful sales effort, but, right? Especially the CEO of a software company. It's a lot like, Uh, that means, uh, an allocation for, uh, usually that had been scheduled for somebody blew up, and there's some thousands of, of chips that are at the market. I think if you talk to the hyperscalers, uh, they will tell you they don't have enough compute and they, they don't have enough space, they don't have enough power to, to, to, to run at all. And so I, I think at the guys who are consuming big blocks. There remains a voracious demand at partners like, like, like our partner in the UAE who's building a sovereign cloud. As we see sovereign clouds emerging around the world in the Middle East and in Europe, our customers like Aleph Alpha, customers in Singapore and and, and elsewhere, there remains this huge sucking sound for AI compute. And I, I think, uh, we're just seeing it sort of move into, uh, sort of the big financial institutions. Yeah, right. And they're, they're, you know, at first senior guys bang their fists, we need AI, and then there's a bunch of, of, of VPs and directors running around and That was last year and, and now it's like, oh, it, it can save a ton of money here and we can create value with it here. And here's how we have to implement it. And this is what we need to buy. And the, what, what was discussion and PowerPoint last year are projects and, uh, value this year. And I think that's the, that the, we'll see that trend for the next 2 or 3 years.

I like your positivity on this market, Andrew. I'm not sure that I, um, you don't share it. I don't know if I share.

Well, I, I think normally I'm arguing software tools guy that the software market's gonna fall apart and it's a bad time to be in software in AI tools.

And my, I'm, my honest take is I feel a little nervous about, um, like the, the, I think the financial services market, you know, we are seeing a lot of increased demand for sure. I feel like the ROI on a lot of these projects makes me a little nervous right now. I guess what I would say from my perspective, it seems like in a lot of places that we're looking, there's a lot of projects where you go, hmm, is there really ROI here? And are projects just sort of getting funded, you know, because they have an AI, um, you know, component? I really don't want to call any customers that might be listening to this podcast, but I think we can all agree, um, that this happens from time to time.

No, look, there's some projects, just like there's some startups where you dust it off a business plan, you, you, you took it out of your drawer, you dusted it off, and you, you added AI in 11 places, and now it's an AI plan. I, I think those are destined not to be successful. But, um, I think the following: I think the companies that begin doing thoughtful projects, um, will build a muscle of deploying and using AI on projects. And I, I think there is a muscle there, right? This isn't one of those things where, uh, and it's probably the same in, in SaaS software too, is if you just throw it up and use it, it's not nearly as efficient as if you— right, you look at something like NetSuite or all these tools that we use, the, the, the range of value you can get from it is ranges from, oh, a good ledger account— accountant could have done that in a leather-bound volume, right, to all the way to the other end as This is a tool that really helps us run a business very differently than we otherwise would. Right? And that is not the tool. That's how the tool's deployed.

Totally.

Right? And I'm— it's, you know, you, you've seen it. Sometimes you, you, your customer success team is working with your customers saying, guys, if you do it this way, this software can really help you differently. Right? And it's the same with AI. That there are going to be some projects that fail, but that the guys who deployed it step back and say, whoa, next time we're going to do this and this and this. And when they go the next time, it's going to crush it. And so I think being on the sideline is waiting until everything is sort of— everything is smooth and perfect in a rapidly changing environment with huge returns to learning. Is really bad strategy. And if there are returns to learning, you need to be doing projects. And they don't have to be wildly expensive projects. They don't have to be, but you have to be engaged with the technology. You have to be trying to solve problems with the technology and with your data. So that's the observation. And if if you haven't thought, even if by doing your AI model, your AI model, the first one fails, doesn't produce value, but instead what you learn is your data's a mess, right? Which is just about what everybody learns when they start to do real AI, is that their data is much less clean than they thought. Um, even if you learn that and you immediately start to clean and organize and marshal resources to make your data better, uh, that's a win because next time, whether it's AI or something else, your data will be in much, much better shape. And so I, I think that those who do will be ahead for sure in this space. Is that, is that in line with what you've seen?

Yeah, I mean, I think that's the positive spin on it. I, I think inevitably, right, the— in the— I think it's a good point. You need to be out there you need to be learning and need to be working with customers. And I agree, long term there's a ton of value here. I guess we've both gone through like boom and bust cycles and it's incredibly hard to, to time these things. But I think there's no doubt there's a fair amount of, you know, excitement and a lot of it's warranted. But I think, I think a lot of companies right now are kind of struggling to turn like that very evocative technology into like something that's like, you know, clearly working on a repeatable basis for, for use cases they care about. And I think it's most extreme in my opinion right now in financial services where there's like, there's so many use cases that really should work and each one is kind of tricky to, you know, to get just right.

You know, there's a famous paper by an economist called Paul David, title is something like The Dynamo and the Something or Other. He's an economic historian and he studied the adoption of electricity in the manufacturing sector. And he did this because in the mid-'80s, somebody asked this question, which was as follows. It says, we see computers everywhere, but not in the productivity statistics, right? And so the question was, everybody's got a computer on their desk, we see them everywhere. But we can't find gains in productivity in any data. And what he showed was sort of what has now become common sort of received wisdom was that at first the technology is brought in to do what's already being done a little bit better. So if you think of the computer, you know, it printed instead of typewrite, it provided word processing, it provided the services of general ledger. And all these other things. But we were pretty good at those things before, right? And so the gains were modest. But when we reorganized our inputs, when we went to the cloud, we unlocked an entirely new way for software to be used. And you saw these massive jumps in productivity. And I think there's some truth to that too, that as we replace things we're already pretty good at with AI, will reduce failures. We will get a little bit better. And the productivity statistics will be disappointing, right? They will, they'll be disappointing. But when we reorganize more fundamentally and really use what is at the essence of AI, I think you'll see massive productivity jumps. And those will be captured in lower costs and new services and cool, cool applications to, to, to make people's lives better. And I, I think that's sort of the, the path. You know, if, if at first we, we use a medical assistant just to double-check doctor's work so that doctors have a failure rate, not unlike other professionals, right? They have, um, if we use it to check work, that's great that we have fewer failures. If we can put a flow together that catches things earlier, um, now we have a better chance of, of, of, uh, healing or, uh, bringing the goal of health to the customer. So I think that's sort of the way I think about it, that you're right. A lot of the ordinary projects will produce ordinary results at first. But that's the way technology's been adopted historically. Right?

That's a good historical reference. I'm excited to—

No, I think that—

Yeah.

Yeah. No, the paper's really interesting. He showed that, you know, at first, They brought in power and they just used it as a backup for belt-driven systems. And it wasn't until after World War II that they said, holy cow, we can reorganize the shop floor now that we have power, the machines are more reliable. And you saw a huge spike in productivity. And I think it's sort of the same with the things we do, is that a little better HR, right? A little better reading of resumes is a reduction in cost, but you're not going to get this massive jump in in efficiency. But if we rethink the way our organizations operate, if we, you know, then you can get a major jump.

Okay, so taking things apart.

I think that's sort of what we've seen in the adoption of big technology.

I mean, pulling out the economics political science background.

I did that Specifically because you ridiculed my undergraduate education. Snark here. Um, I mean, come on. We had to read books at Stanford in political science. I know that your engineering degree didn't have a lot of reading of books.

I think they forced me to take a humanities class that vaguely remembered European history from 1840 to World War I.

Right. That's right. That's good stuff.

Um, did you, um, okay, but I do want to ask you this. So, sure, when, um, I feel like, you know, your, your business is really, or the inference business at least, is a real bet on, um, kind of open source software, which is kind of an interesting question. I've kind of noticed that every CEO betting on open source software seems to really believe in open source software definitely winning. Yeah, from my perspective, it seems a little less clear, like, what Winds, like, how— what is your feel? Are you, are you sure that open source models will be the future?

I'm not sure. Um, I think, uh, we, we today have customers that are closed source and are running, uh, inference on our, on our equipment. Um, we, we don't, uh, depend on open source. I, I think there is a, a battle Uh, and it's unclear who will win, and it's unclear who's won in other open-source, closed-source battles, right? Um, I, I think the number of large enterprises that run true open-source OSes as opposed to, uh, a supported version of the open-source OS, right? There are lots of places you can look. In other areas, open source has crushed it. But I, I think, uh, it is not clear, it is not obvious. Um, but I, I think it is healthy for the ecosystem and it's good for everybody that there is that this unbelievable sort of Cambrian explosion of innovation and that it's not just one or two companies and it's not just those you expect, right? I mean, The leading, a different way to say it, Lucas, is the leading closed source company, right, was a startup nobody heard of 7 years ago, right? And that's an amazing statement, right? That companies that had been around 40 or 50 or 60 years aren't the leaders in model building. That, you know, Facebook has dedicated itself to open source and that's a, producing unbelievably good models. Others have teams of 20 or 30 in France are producing extraordinary models, right? I mean, you, you look around and it's— there are lots of paths to do really interesting work here. And, and so I, I'm not sure, uh, what the strategy is, or whether the strategy will be some sort of hybrid where you begin on top of Llama and you, you, you take some do some additional work there, or you pair open source with some closed source, um, uh, uh, to do your prefetch or some portion of the work, um, or you write glue software that links together hundreds of smaller open source models, and that glue software is still extremely complicated and provides a ton of value, right? And, and so all of these are I mean, one of the fun parts about this industry is that it's, it's not storage. It's not sort of stepping along step by step. It's dynamic. It's changing. Uh, there are lots of ways to attack the opportunity and you're seeing many different strategies play out and that's really fun.

Yeah.

Um, it's sort of even, even when there are other people's strategies and you sit back and you say, this company, wow, they're, they're, They're going after giant models with a collection of small models. All right, that, that's an interesting thing to watch, right? Uh, a confederacy of small models. Um, these other guys are— some other guys are just, we've got advantages in money, let's spend as much as we can to buy as much compute as we can, and we believe that whoever has the biggest compute will win. That's a different strategy. And watching all of these unfold is, is like like a video game. It's really fun.

Which company would you describe as that strategy? If we have more money, so we're just going to buy the most compute. Who are you thinking of there?

I think OpenAI has stated that they believe that having the biggest compute is one of the foundational elements of winning. I think Elon's strategy of trying to set up some of the biggest training clusters on earth is that strategy. It may well be the right one. I don't know. We have yet to see.

All right. Getting some thoughtful takes, but not some hot takes. Let me—

Thank you. All this media coaching is working. It's inducing ridicule.

That's that. Goddamn it.

Um, should we talk a little bit? I mean, come on.

No, no, I got, I got a couple more that I really want to get to. So, um, yeah, one thing we didn't talk about last time is, um, is kind of AI applied to, um, actually manufacturing chips. And we've been hearing a lot about that, like a bunch of guests have come on and talked about that. Like, how, how relevant is like recent advances in AI to, um, your actual chip making? Is that part of what's made you successful, or is that sort of separate?

So, uh, the EDA tools that are fundamental to chip design have used ML, uh, for a long time. They've used all sorts of statistical methodologies. They've used trees. They've used all sorts of optimization theory, uh, of methodologies for a long time. Now, they haven't used what, what is sexy now in AI, uh, the generative models and, and those. Um, I think it is still very early, and as is often the case when a new technology is forced to compete with a very mature technology, uh, the guys are pretty darn good at designing transistors and circuits right now, right? I mean, circuit design is pretty, pretty advanced. Now, I, I don't think we will see machines designing, uh, uh, chips, uh, or AIs designing chips that are better than, than people for 3 to 5 years. But I, I think there are all sorts of parts of, of every stage of the chip design that could be improved and time rung out. Right? This isn't replacing insight. There are repetitive tasks in, uh, place and route. There are repetitive tasks in timing closure that, that, uh, machines ought to be better than people at. And so I, I think, uh, what will happen is that, that, that the AI will be used around, uh, the, the core insight over time, and that will make chip design lower cost and faster. And then it will nibble its way in over time into some of the, the hardest of problems, which is converting the, uh, the desire of the chip architect into logic.

But it sounds like you inside your company aren't are doing a lot of this.

No, we're not.

Your bet is like, we're gonna make a giant chip and that's the, that's the—

well, I, I think something different. I think, um, you need a huge amount of data across many, many chips to do this, and the people who have that data are TSMC and Samsung, or they are the EDA toolmakers. Right? You want to play where you can win. And, you know, what you'd like to have is insight over 500 or 1,000 chip designs. Right? To do real work, not to do marketing. Everybody can say, oh, I'm using ML and AI here and there. But, right? What you'd like is the database describing the logic of maybe every chip ever made.

Mm-hmm.

With that, I think a lot of it, right? But who has that? Well, uh, nobody. Now AMD has what they've made and, and NVIDIA has what they've made and Intel has what they've made. But, um, I think the assembly, and this is a, a general problem for where data is not easy to aggregate, um, I think that delays the, uh, use of data-intensive technologies like, like AI.

Totally.

Now, it doesn't delay the marketing, and it doesn't delay some of the, the, the wrapper, right? But, but the core insight— we, we've got some time, uh, before it's better than, uh, you know, engineers with 30 or 40 years of chip design experience.

Awesome. Well, Andrew, thanks so much.

How's your business, Locus? Why don't we give you a 2-minute commercial?

It's a—

would have been better if you—

well, actually, we should totally integrate our— there's a really natural integration that we should do with you before we launch this. Like, we now have like an LM evaluation observability tool, so we can—

oh, cool.

And it's really popular. So it's—

so the perfect example, already a customer of yours.

Yeah.

And, uh, I, I think, uh, this is a perfect example of what I was saying earlier, was that, um, AI will behave just like this, where, uh, people will, will sort of learn that they're not using it as well as they could use it, right? I mean, I think that's exactly right, and that's no different than, than other cool tools. And so I'm sure there's a ton of stuff in your, in your software stack that we haven't touched yet and that could be valuable to us.

Totally. But yeah, we should actually ship this with, we can do a, we have a really nice kind of evaluation system that we could run on you, you know, against like other, other stuff and sort of show people like, you know, third-party assessment of like throughput and accuracy and speed on a bunch of benchmarks. And we can also set it up so that, you know, your customers could do that themselves.

Oh, that'd be great.

Which would be great because then it would be easy for them, you know, to just see the value, the economic value. Because if we actually get— if you give us your pricing, you know, for people that give the pricing, we can actually put the pricing right in the application. Then people could really starkly see the difference.

Um, that would be really cool.

Yeah. Yeah. We should get, if you could give us like whenever you can, like some preview access to it, we can sync it up so it automatically, uh, we get you on today. Oh, perfect. Let's, yeah, let's do that.

Our engineers love this and you know, they'll play around and I think we'll blow their mind.

I think, nice.

Um, you know, we're getting quotes from, from engineers that just unbelievably rewarding and you know how nice it is. When you've spent years of your life building something.

Totally.

And someone sends you 3 lines like, this is really fast, right?

Totally.

You, you come home to your wife and you go, what, today was an okay day. It's like, why? And you list 8 things that went wrong, but I got an email from a guy I've never met who used our, our, our equipment and this is what he sent. And your wife looks at you like you're outta your mind. Um, yeah. Um, and, and so when you get those, Um, you, you, you feel good and, and we're getting those just raining down on us.

That's awesome.

So that, that feels awesome. And I know we'll, uh, you guys will be really fired up.

Yeah.

So look, cool. Um, thank you for having us on to talk about our, our, our new inference capabilities.

Yeah, absolutely. Excited.

We really appreciate it. And it's always fun to talk to you, Lucas. And, uh, it was lovely to see your family in, in June. Um, that was really, that Uh, to see the little, the little people running around, that was great.

Totally. It's pretty fun. A 4-year-old with an LM is a pretty hilarious, uh, thing, I'll tell you.

Yeah, yeah, for sure, for sure.

Uh, yeah, great to see you, Andrew.

Thank you. You be well.

Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for for future episodes.