#245 Can We Make Generative AI Cheaper? With Natalia Vassilieva, Senior VP & Field CTO & Andy Hock, VP, Product & Strategy at Cerebras Systems

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI Transformation Journey on datacamp.com.

Cost pressure I think is going to drive innovation. But second, I think we as a community are really using in some sense a lot of the traditional AI work that's been done today has been done on traditional general purpose infrastructure. Those processors are suitable, but they're not optimal for this work. At Cerebras, we've built a completely new class of processor that's built from the ground up for AI Foreign.

Welcome to Data Framed. This is Richie Generative AI is amazing, but as projects are being put into production, some grumbles emerge around performance. It's often too slow and too expensive. As user expectations around AI reasoning and accuracy increase, and as more complex use cases like AI agents become more prevalent, the challenges are only going to get trickier. Some of the solutions to this around model design and software, but today's guests have taken a deeper approach. They hail from Cerebras Systems, which has an AI inference platform based on custom chips. So rather than running AI on top of general purpose GPUs, the hardware is designed specifically for generative AI. I'm keen to find out about the latest developments in the AI tech stack on both the hardware and software side, as well as the implications for organizations trying to create AI applications. My first guest is Natalia Vasilieva, the Vice President and Field CTO of Machine Learning at Cerebras. She's an expert in natural language processing, computer vision, machine learning and information retrieval, and as field cto she helps drive product adoption and customer engagement for the wafer scale chips that Cerebra System provides. Previously, Natalia was a Senior Research Manager at HP Labs, leading the Software and AI group, and she was also an Associate professor at St. Petersburg State University. Next up is Andy Hock, so Cerebras Senior Vice President of Product and Strategy. That is, he runs the product strategy and roadmap, focusing on integrated AI research, hardware and software to accelerate the development and deployment of AI models. He's got 20 years of experience in product management, technical program management, enterprise business development, algorithm development and data analysis for image processing. And previously he was product management lead for data and analytics for Terra Bella at Google, where he led the development of machine learning powered data products from satellite imagery. And earlier he was the Senior Director for Advanced technology programs at Skybox Imaging. So let's find out about the cutting edge of a.

Hi Natalia and Andy, thank you for joining me on the show.

Happy to be here. Thanks Richie.

Excellent. So we've seen a ton of, well, quite frankly dramatic progress around generative AI in the last six years. Do you see this dramatic progress continuing much longer? Natalia, do you want to take this?

Oh yeah, I think we are just starting then. There is still a ton of both advancements in fundamentals, how generative AI models can be trained, how we can make them more efficient. But also I think we're just tapping into what those models can do. There is a lot of excitement these days around, I'd say, educational and entertainment use cases. So all of us probably have tried using ChatGPT, looked at image generative models. But in terms of how all of those models can be realized in real life use cases, I think there is still ton of discovery to be made and it will drive requirements to the models and it will require it will drive a lot of additional progress both in training those models and also on applications.

Okay, that's interesting. So it's really going to be the wider adoption of use cases that's going to drive the technological advancement. And Andy, do you have a similar take on this or have you got a different opinion on how progress is going to go?

Very similar take. I think we're just scratching the surface. I found myself reflecting as Natalia was responding. Imagine where we were with say, computer graphics 20 years ago and where we are today with rendering and gaming. I think we are today with AI, maybe where we were with graphics or analytics 20 years ago. I think there's a huge proliferation of applications brewing right now in both enterprise and research. And I think we're just at the beginning of understanding which ML techniques and network architectures and data types are the most amenable. So I'm thrilled to be where we're at, particularly at Cerebras on the AI computing side. But I'm really looking forward to where the field is going over the next one to two to 10 to 20 years.

That gaming analogy is kind of cool. It's like we've just reached the sort of PlayStation One Xbox era and it's like, wow, stuff's in 3D and it looks really cool, but like, you can't imagine what, you know, PlayStation 5 looks like.

Yeah, absolutely. You know, we make this analogy pretty frequently, but you know, Cerebras is well known as an AI computing company. We spent a lot of our time focusing initially on software support on our hardware systems for training, but particularly around large model inference. We make this analogy that I think we're in some sense in like the web dial up era of inference in particular, where today we're in that, the moral equivalent of the dial up era, where we're waiting for the modem to dial up before we get our answer, but we're not yet in this, in the modern day network equivalent of like instant responses. So I think both in terms of the technology and the applications, both on the training and compute and inference side, we're just scratching the surface, man.

Okay, yeah, I think nobody really misses that kind of that modem dial up noise where you're waiting for things to load.

Exactly, exactly.

Hopefully we get broadband soon.

That's right, it's coming.

Nice. So lots to be excited about. And I think one of the caveats is that as a lot of generative AI is starting to get pretty put into production, people are kind of going, well, actually it's kind of expensive at the moment. There's a lot of costs around just running these models, doing inference beyond trying to train things yourself. And so do you think these costs are going to affect the development of future LLMs? I'm happy to.

So I think, look, a couple of things are going to happen. First of all, cost pressure I think is going to drive innovation. But second, I think in some sense a lot of the traditional AI work that's been done today has been done on traditional general purpose infrastructure. And that's one of the big cost factors in innovation. Right. So people are building models on large clusters of small general purpose processors. Those processors are suitable, but they're not optimal for this work. At Cerebras, we've built a completely new class of processor that's built from the ground up for AI, and as a result it's faster, but it's also more efficient. So that means that if you're say, training a model, you can go an order of magnitude faster than you could on an equivalent GPU system. And it's going to be more efficient, not only in terms of developer iteration, being able to test more ideas per unit time, but it's also going to be more power efficient underneath the hood. And then I think given that right Infrastructure for AI research. Researchers are going to build not just larger models, but they're going to build models that are more efficient and use more efficient methods like sparsity. And given the right processor, you can harness those advantages and turn those natural advantages, say a sparsity in a neural network, into performance advantage and cost savings. And a practical example of this, and it's something that we're pretty excited about at Cerebras in the very near future, is AI purpose built hardware for inference. And we're launching inference. And just as a data point, right, our inference is going to be 20 times faster than GPU opening up new applications, but it's only going to be about a third of the cost. Given the right infrastructure and new methods. I think cost pressure is going to drive innovation, but the right infrastructure is also going to make it more accessible.

Okay, so that's cool that you're not just going for, let's try and help the developers. It's also going to be like, let's bring this to a sort of wider audience. Now it seemed like there's sort of two levels here. So you've got the hardware level. You talked about having custom chips rather than GPUs, and then also some of the software techniques. I'd love to get into both of these. So maybe we go with hardware first. So what makes these chips different from a gpu?

At Cerebras, we approached the problem of AI compute from first principles. And what I mean by that is we actually looked at the workload of neural network training and inference and we asked, okay, what kind of compute operations, what kind of precision, what kind of communication memory access patterns does this work have? And we built our processor around that. So obviously this chip is big. Most computer chips are the size of a fingernail or postage stamp. The chip that powers our system is the size of a dinner plate. We call it the wafer scale engine because it's the scale of a full silicon wafer in chip fabs, it's about eight and a half by eight and a half inches on an edge. So how is this different from a GPU is the question. Right. And it's not just because it's bigger. AI compute requires massive computation, sparse tensor based linear algebra operations and the ability to handle data in motion. So high communication bandwidth and high memory bandwidth. And that's why we built our wafer scale engine the way we did. And it's what the wafer scale engine is best at. So this has 900,000 cores on one chip. They're all designed from the ground up for sparse tensor based linear algebra operations, mostly at lower precision, because that's what AI compute wants. And all those cores are directly connected to one another on the wafer, and there's 44 gigabytes of all on chip SRAM like a L1 cache. So lots of memory close to computer and the compute can talk to each other very quickly. So all those attributes of AI computing that I talked about, that's what this thing is built for. And it's fundamentally different than a legacy general purpose processor that's small and far away from memory like a gpu.

Okay, so really, because you're not having to solve many different problems like a general purpose GPU is, you can focus on just optimizing for just these sort of linear algebra operations that are needed by the neural network.

Bingo. Yeah. And the result is, you know, order of magnitude faster on both training and inference, depending on the workload, and more efficient and easier to program. And that trilogy of benefits all acts to accelerate innovation and accelerate compute on the inference side for models in production.

So you mentioned a few sort of ideas around how large language models can be done better from a sort of design like the neural network architecture point of view. So maybe let's get into some of those. The one you mentioned first was sparsity. So Natalia, can you talk me through what is sparsity in a neural network and why would you want it?

In general, sparsity is when you don't have all of your weights being non zeros or activations. So sparsity is when some of the elements of your matrices are zeros, doesn't matter which matrices are layer. So when we're talking about sparsity as it's applicable to training of neural net, how it can help you, it can reduce computational cost and it can reduce also memory requirements to run inference or to train these models. I think overall in community there is a broad acknowledgment that sparsefet is one of their avenues to make training more efficient, to make our models less expensive to train and to run inference with. And there are several approaches how you can implement that during your training or during inference. So you can think about activation sparsity essentially when you train a neural network, that's always interaction between the parameters of the models, the weights, that's one of the matrices that participates in this computation. And data or activations, that's the second argument, and they combine together and you do essentially multiplications on the linear operations on top of that. So you can think about inducing sparsity or making sparse either one element or another. And you can take advantage when you do either of those. There are many methods there which allow you to again reduce cost of compute and reduce memory footprint. If you induce zeros in your model parameters, bottom line, throw some zeros either into your weights or into your activations. With that, reduce the cost of computation and reduce the memory footprint.

So I guess the idea is that the weight of each of the neurons in the neural network, that's some sort of measure of its importance. And if it's sort of quite small, you can just pretend it's a zero and that lets you ignore it effectively in some calculations. Is that right?

Yeah.

If you think about the whole process of training neural network, you're starting with the huge pool of parameters. And it's very well known that all of the current neural networks over parameterized. So we throw way more parameters than it's needed to actually solve the problem. And then the goal of the training is to understand which weights should be large. So essentially which weights are meaningful and which weights are small, they don't bring any value. So just if you can figure out how to do not train those smaller weights which don't bring any value from the get go, you would be better off.

So it's just a way of simplifying the computations for bits of the neural network that you don't need, you just sort of ignore them. And then the other aspect was about using sort of smaller numbers, this idea of like quantization. So can you just talk me through again, what does that mean and how does it work?

Yeah, that's another approach. How you can reduce both compute cost and memory footprint for your model is when you use lower precision to represent either weights or activations in your model. If you recall your course for computer science, every single number that you can represent and the computer can read can be represented at different precision and can take a certain number of bits. So you have single precision, double precision, half precision, four bit precision, two bit precision, maybe binary, and you just have zeros and ones and less precision, less precise your computation is. So with the numbers with less precision you can represent smaller range, so you can do only certain number of numbers and also smaller precision. So how much resolution you get for each of them if you go to lower precision, again it takes less space in your memory and it's also less expensive to compute, but it's less precise. So there are a lot of approaches how to make training of neural networks work with lower precision. Today, when you train large language models, you typically do that in mixed precision. People typically say half precision, but in reality you still have to have some operations in single precision, 32 bits. But there is a lot of research going on how to make it stable at low precision. And people have figured out much better how to reduce the precision of weights after the model has been trained. So let's say I've trained my models, I have all my weights and have precision. Now it's much more clear how to reduce precision of those trained weights without huge impact on accuracy and allow cheaper serving and cheaper inference.

Okay, so that's interesting. So you train with bigger numbers, these half precision numbers, and then afterwards you say, okay, well maybe some of these values, we can sort of shrink them even more because you don't care about the precision in this area of the neural network.

Yeah. One additional caveat is to actually take full advantage of all of those. From the theoretical algorithmic point of view. Sparsity is great, quantization is great, but to take advantage of all of those in real life, you need to have also hardware which is capable to work with low precision numbers and the hardware which is capable also to work with those bartenders. And that's a very also important notion to keep in mind.

Okay, interesting. Yeah. So you need to make sure the hardware supports what you're doing with this. And Andy, I think you mentioned before that the Cerebras chips, they're designed to work with lower precision numbers, is that correct? So it's designed especially for this kind of neural network problem.

That's exactly right. So as Nataya mentioned, with Sparsity, most of the data and most of the trained models that exist have a lot of zeros in them. And on a traditional compute platform like a CPU or a gpu, when you run the model, and when you run the model, usually what you're doing is you're doing multiplications and additions. And a traditional dense compute platform, like a CPU or GPU in particular GPU will take that zero value and will actually execute the multiply by zero operation. So it takes time, it takes compute cycles, it takes power, but we actually know what the answer to multiply by zero is. And so if you had instead a hardware platform that could detect those zero values and just avoid them and move on to the next non zero value, you would end up accelerating the compute and saving yourself power. And that's what the Cerebras platform does. So we sort of acknowledged in the design of our hardware that the nature of the compute workload involved a lot of zero values. And so we built that in so that we can actually take advantage of that and our users can take advantage of that natural aspect of the data and neural networks. Similarly, on the, on the precision side, we support in our hardware today 32 bit, 16 bit and 8 bit numeric formats. We're really in the sweet spot of the numeric representations that are common for both training and for inference. And we can execute those at massive performance advantage over traditional machines on both the training and the inference side.

So yeah, it's very convenient that you multiply any number by zero, the answer is going to be zero, and you can do a little shortcut there. Okay, nice. So I'm curious, are there any other techniques that are used in order to make large language models more more efficient or more performant?

Yeah, I think we've discussed a couple of those. So using sparsity and the different types of sparsity. So remember I talked that you can introduce those zeros either into weights or interactivations, just activation sparsity or weight sparsity. There are also different techniques on how regular those zeros are. Among the ways so we can talk about structured and unstructured sparsity. Maybe some of you have heard about models called the mixture of experts. That's a perfect example of models with unstructured block sparsity. Yeah, I think for all of those techniques, the goal is that if I have a certain budget of compute, how can I get to the best, accurate, the best, highest quality model given that amount of compute? And it's been shown that with sparse models, you can get actually to better models by spending the same amount of compute compared to what you would potentially get with dense models. Mixture of experts is one example of those structured sparse models.

Yeah. Do you want to just talk us through like how mixture of experts works?

Mixture of experts. When you hear mixture of experts, maybe you think about the common approaches which have been in past called ensembles of experts. When you have completely separate models, you have completely different experts or predictors, and then you take a vote and decide what the accurate answer is. Mixture of expert model is not exactly that. So you still have those experts, but they are built in into single analytical model. And from the computational point of view, when you train a model, usually with a dense model, you have every single data point to influence every single weight in your model. So you need to make computation and see how those two react to each other for every single pair. For mixture of experts, it's an example of weight sparse models. So you still feed all of your data points through your model, but you do not get all of the model weights interact with all of the data points in your training data set. So the model selects, okay, this batch of data, this number of training examples, they will only influence this partition of the model. And that next set of examples will influence and will change the values only of that partition of the model. With that you can remove this all to all dependency and on the subset of the weights get to be influenced by certain data points and that what's reduced the computational cost at the end.

Okay, so I guess rather than having one giant large language model, if giant and large go well together, I'm not sure, but you have a few smaller models that are essentially, I guess, glued together. Does that sound about right?

No, no, I think it's. You still have this large and giant model, but you don't have all parts of the model to take inputs from all of the data points. So the model is still the same, but the model decides inside that model. Every single part of the model will look only into some parts of the data. So not all parts of the model are created equal, I guess. And that's what makes why people call it mixture of experts, because you can think those parts of the model being different experts and they only responsible for reacting to certain data points that make sense.

Okay, so you've got one model, but just different parts of the model are responding to different kinds of inputs.

Yes, to different kind of inputs. Yes, that's exactly right.

Yeah. I'm going to go out on a limb here because I'm really enjoying this conversation, but I'm going to go out on a limb with a really, potentially a really bad analogy. But I keep thinking in my head of an octopus and I can imagine a prompt, an input comes into the octopus and comes into the head and the brain and, and that does something. And maybe it only needs one tentacle to respond. Right. And so input comes in, only needs. Even though it's got eight tentacles, the model is still large to your question, Richie. Right, but it only, maybe a particular input only requires a part, a single part of the model to respond. And then a different input comes in and requires a different tentacle of the model or a different expert in the model sense to respond to it. And therefore when you're training, you're still training a large model, different tentacles, but at inference time, maybe you don't need all the experts to respond, you only need a fraction of the experts. And therefore the total computational cost at inference time is Significantly lower.

I really like that analogy. Yeah.

So I think maybe you're being gracious. Right.

All right. Mixture OCTOPUS models is the next generation.

I'm going to pay for that, man. All right.

So one other aspect of at least creating large language models that seems expensive is that after you've trained it, you need to get some sort of feedback from humans on whether or not it performs well. So it's doing this reinforcement learning step and whenever you involve humans, I guess things get expensive. So is that technique going to stay around for a while or are there ways to avoid having to use humans?

There are several alternatives to that, although I think the jury is still out how accurate they could be. So people use. There's a notion of RL AIF when you use not human feedback, but you use AI feedback. So if you have a smart model which has been already trained, you can use that to help you to align your new model. There are also techniques like direct preference optimization, which doesn't require human feedback either. So I think community is looking for other ways how to avoid or minimize involvement of human feedback. But jury is still out there which ones are the best and if we can get away with without any human.

Feedback at all, I'll say maybe more generally. Right. Like reinforcement learning with human feedback is basically one way to get a pre trained model to behave the way that a developer wants it to based on some optimization function. We're all building models to try to get them to do something. And currently there's a pretty wide portfolio of methods that a developer can use to help the model achieve better output. Right. More accuracy for a particular task. Right. So there's methods like fine tuning, there's methods like reinforcement learning with human feedback. There's methods like DPO and rlaif like Natalia mentioned, there's also building models that are augmented by specific databases, retrieval augmented generation or RAG models. So a bunch of different ways that people can take a model and basically help it be more accurate for a particular task. And they don't all require humans, but a lot of the models are serving functions for humans. So often when you're sort of measuring a model's performance, you want that human feedback in there. But certainly there are a lot of methods that don't require sort of rich, interactive, time and time intensive human feedback like rlh. And coming back to this discussion about compute, right? Like if you're building a new model, which recipe you choose or which method you choose to both train your model and then make it work really well for your application, it's not as though there is a specification someplace that just tells you if you're an enterprise application developer and ML researcher that like, you know, if you have problem X, use method Y or collection of methods Y to build this model. We're really learning that as a community as we go right now. And I think one of the things that's really powerful about having the right high performance AI computing platform underneath, as Natalia and her team who do solutions engineering with some of our customers at Cerebras know very well, is the value of that high performance engine underneath the hood is that you can try a lot of different methods, right? You can try training a model from scratch, or you can try taking an open source model and fine tuning it. You can try doing other methods like retrieval, augmented generation or RAG or dpo. Right? You can, you can try these and you can iterate very quickly because you have a fast engine underneath the hood, even for these compute intensive problems. What Natalia and her team has seen with our customers is that this combination of sort of new methods and wanting to build models that are accurate for a particular set of domain specific tasks is being able to iterate really quickly, both at large scale and small scale, with different data representations and different techniques, and quickly move not to just building a model, but building the right model for your application.

Okay. Yeah, So I guess it sounds like building a great large language model or probably anything with AI is very much an art form. And so rather than just like a procedural thing. So he's got to have that agile workflow in order to test a lot of different things in order to get something. That's great.

Yeah. And we were, we were chatting a little bit about this before. I mean, it's definitely an art in the sense that it's evolving and I think involves a lot of creativity, but it's actually becoming very much a scientific process. Right. So testing something and then learning from that test, developing hypothesis, testing it and going back. So being able to quickly iterate, like I was mentioning, with different data sets or different hyper parameter settings or large models versus small models, that's I think what gets people to really proficient and accurate models for their specific application. But that's hard on traditional systems.

Yes, definitely. If you happen to wait for hours or days to get the results of one sort of experiment because things are running too slowly, then it's not going to encourage you to do lots of different experiments.

Exactly. And it was, I often joke about this, but if it takes you a couple of days or a week to train your model in grad school, maybe that's okay. Right, because then you go for a ski weekend or maybe take a little vacay or work on another project. But if you're building a model to help predict optimal drug combinations for cancer, or if you are building a model to help say, detect fraud in time series data for financial transactions, or if you're building the next great chatbot to improve customer service or retail engagements and that's part of your core business, waiting doesn't work. And so the development velocity is really important. I think that's what's really driving the field forward. And that's back to your first question. I think why we're really excited about pushing into the next 1, 2, 10, 20 years and seeing what people will build given the right tools.

Absolutely. And so I'd like to talk a bit about this sort of business items you mentioned. Time to value is incredibly important. I guess like the trillion dollar question is like, how do I do generative AI cheaper? I think there's a lot of chief financial officers pushing for this right now. So I guess maybe we'll start with companies that have some kind of generative AI in their products and they're doing inference. So what are the sort of main things companies can do in order to reduce the cost of inference?

I think to start building the right model, there are ways that given a particular inference platform or given a particular performance target, you could choose to train a larger model or a smaller model that will have say, higher accuracy or lower accuracy, but also higher cost or lower cost. And so step one is think about the business problem that you're trying to address. Think about what the performance metrics are that matter in terms of say, responsivity or latency throughput, how quickly a model can answer questions, or how quickly your model needs to answer questions and how accurate it needs to be, and then build the right model for just that. I think what we're seeing right now is a lot of enterprises are starting on their generative AI journey by just taking say a general purpose pre trained model from open source and like giving a shot and giving a shot with that, or maybe trying a third party proprietary model that's available through a service and giving that a shot. And they're using that to test the value hypothesis and figure out like if a generative AI solution will help them drive revenue or drive efficiency. But then after they get past that initial value testing process, what we often see at Cerebrus is enterprises coming and saying, yeah, we tried a couple models and now we would like to build a model that's purpose built for our application. You could still start with an open source model, but maybe fine tune it on my own company data so that the model is really tuned for my application. And we work with enterprise customers regularly every day to help them build models that are purpose built for their application, including thinking about the right size to optimize inference performance and cost in production. That was a long winded explanation of step one. Build the right model that fits your application and your inference performance and cost envelope. The second step of course, is once you have a model that works, deploy it on an inference platform that delivers the performance that you need inside the cost envelope that you're targeting. And as I mentioned earlier, we're really excited at Cerebras to be announcing very soon the launch of inference on our system still uses the big chip, right? So same hardware. In fact, all the attributes that I talked about about this chip earlier that benefit it for AI training are also benefits that unlock literally GPU impossible performance at inference time. So we're going to be launching inference on our systems very soon and it's going to be 20 times faster than GPU and at about a third of the cost. So build the right model and then deploy it on the right hardware.

When you say it like that, it sounds so easy. Build the right model.

It's been a great chat. Thank you, Richie. Yeah, it's definitely not that easy. And Natalia will definitely, definitely know this better than I do.

Yeah, I just wanted to kind of not only build the right model and use the right hardware for the inference, but also decide from the get go which hardware you're going to use for inference. If you want at the end of the day to optimize for as cheap or as less expensive inference as possible, that's likely also mean that you will have to invest more into training, but you can save later on.

In France, Andy mentioned that sometimes like you're going to care about different metrics. So sometimes you need something that's really low latency so you get quick responses. Sometimes you're going to care about high performance. Can you just give me some examples of like different generative AI applications where you might care about like one thing or the other and like what might have a different price point or how do you go about deciding this sort of stuff?

Yeah, absolutely. So let's talk about latency and, and throughput. Right? Latency is like how long you wait, throughput is how fast do the answers come and maybe just two really simple examples. One example is let's say a conversational chatbot. And there latency and interactivity is quite important, right? Throughput is definitely important too, because how fast say the words print out is important, but latency is very important because you're seeking interactivity. On the other end of the spectrum might be a batch offline use case maybe where somebody has for example, built a very large complex model that does things like try to simulate how well particular candidate drug molecules might work on a particular disease. And they're maybe searching tens of hundred tens or hundreds of thousands or even millions of different candidate molecular drug designs against a particular disease. But they can do it offline, they're not trying to be interactive with it. Their throughput really matters because let's say that it takes me six months on a traditional machine to parse through those hundreds of thousands of compounds. If I could do that not in six months, but in six days or six hours, then my ability to iterate and go faster and find the right solution to my problem becomes dramatically faster. So that's just an example of where like a latency intensive application versus a throughput intensive application. What's becoming really exciting these days though in language models is not just language models that power conversational chat, but language models that power things like software coding and these so called agentic models where you might have a conversational language model front end and then several agent models that go out and perform different tasks for those applications. It's not just latency that matters, it's really overall throughput because those models are processing a lot, a lot of data very quickly to say, generate a new software program or go out and perform some complex multidimensional task with many agents. So with our solution, right, going say 20 times faster on throughput not only opens up far more responsive chat, getting us past this dial up era of Internet into the broadband era of inference AI, but it also opens the door to new language model inference applications beyond just conversational chat to some of those inference applications as well as inference for far larger models like multi hundred billion or even trillion parameter models.

Okay, so I think the term large language model is going to come sort of upside giant language model or something like that, enormous language model. I'm not sure what the term is going to be, but yeah, sounds big.

I think you got it. Enormous. It's pretty good.

Yeah.

Yeah. Okay, so just to summarize what you're saying, I think like the idea is that if you've got chat in your product feature, then you're probably going to care about latency because humans aren't going to wait a long time for a response. And if you're processing lots of data, then throughput is a thing that you care about.

Yeah, maybe another way to think about that. Chat with one single person, how fast the model is responding. Or can you afford chatting in parallel with thousands of people? And what's more important to you, throughput is being able to chat with thousands of people in parallel. Latency that being really quick in responding to one single person. It'd be great if you can do both of those at the same time.

Just thinking about a podcast with a thousand people, that sounds terrifying.

Well, that's why we're going to build a digital twin of you Richie, so you can parallelize yourself and do many podcasts in parallel.

That does sound fun. So I've had previous guests telling me that podcasters are about to become obsolete by AI because we can. I'm going to be replaced by a bot. But I like the idea better. Digital twin of Richie is doing thousands of data frame episodes in parallel.

Well, you can ask digital twin Richie to perform maybe the podcast that you didn't want to do as much, and then you can dive into the ones that you really care about.

I thought you were going into reinforcement learning, where digital twin and actual Richie are talking to each other and getting better and asking questions.

Next episode.

Okay, yeah, yeah, I'll try and throw that together. All right. One more thing I want to touch on is so I came across this idea of the centralization decentralization pendulum, where computing keeps sort of switching backwards and forwards between stuff that's happening in the cloud and stuff that's happening locally on your own compute at the moment. Like a lot of generative AI services, they're cloud based. Do you think that's going to persist or do you think it's going to swing back to having local hardware?

I think it's all very intertwined with the requirements and progress. I don't think on the training side we will move anywhere soon to decentralized training. Yes, I think there are a lot of works on federated learning and how to make it work with a very large network of independent small devices. But all of the models which are trained today, it's not even cloud. It's a very special partition of the cloud. And if we are serious, all of the really fundamental foundational models they are trained on, on PREM hardware or on part of the cloud, which will have been carved out and specially designed, specially connected so you can Think about that as a very, very centralized beast. So I think training will continue to require these a lot of centralized, extreme exascale compute resources for inference. It's more tangible to envision that inference will happen on smaller devices. And we can potentially see that decentralized. And with all of the techniques that we talked about, this like quantization is easier to apply after the model has been trained. Things like distillation, things like speculative decoding, things like there are, I think, many more ways how to reduce the cost of inference and how to make it work on smaller devices compared to training.

Okay, so we're a ways off sort of training GPT5 on our own laptops, but maybe some sort of inference is going to be local.

Yeah. And there may even be some very small scale local learning that happens on a device, like maybe some learning about personal habits or preferences on device. But I think Natalia is exactly right. The compute requirements or sort of primary model training today is happening on huge dedicated clusters that are really built for that purpose.

Before we wrap up, I'd like to know what are you most excited about in the world of AI? Natalia, do you want to go first?

I think I'm the most excited again about all of those use cases which we don't know yet about. And when people ask me where AI can be used, my answer is usually everywhere. Because I can FantaSize application of AI in every vertical, in every industry, everywhere. But to me, the most exciting when it's not a fantasy, but when it's really implemented. And then there are things real which we cannot even imagine today. The richness of applications and what can be done with this technology in the future is the most exciting. And like, yeah, I think, I believe that AI will, will change the world very significantly. It will become either much, much better than it is today or much worse than it is today. I'm an optimist. I hope that we'll be able to handle that and our kids will live in much better world and we'll have things like personalized medicine, will all live longer. We will be working on what we like to do and we won't do any boring tasks anymore.

No more boring tasks. I like that. And yet the goal is widespread adoption of AI and a better world. That does seem very utopian. I like it. Excellent, Andy.

It's hard to top that, right?

I think bring some reality.

Yeah, yeah, look, I think large scale AI has this massive potential to transform the way we make decisions, the way we live and work. One of the cool things about Working with Cerebras. Given the nature of our hardware being the highest performance training and inference systems in the world, customers tend to come to us not with just tiny problems, not wanting to make sort of incremental advances on existing solutions, but they tend to come to us with these really grand and ambitious world changing ideas, right? We have customers like GSK using and building fundamentally new AI models to accelerate drug development for human disease. We have customers like the US Department of Energy who are building large and small models alike that are new foundation models for science to better understand physics and biology, for things like energy generation and the environment. We've got customers and partners like our big strategic partner in the UAE, G42 that are building fundamentally new language models in non English languages that will reach a far broader population of the world and expand the population of inventors and users of AI systems and also building models for say, fundamental new models for medicine and healthcare. So I think this sort of enterprise opportunity to build new models to improve revenue and efficiencies in business are huge and transformative in themselves. But I see this untapped potential that if we give people the right tools and we're cerebrous, we're tool builders, right? We're building computer systems that other people build upon. Given the right tools, I think we have an opportunity to unlock the potential on all those really, really exciting domains. And back to the beginning of our conversation, right? We're scratching the surface today of what people can do. And so I can't wait to see what the next generation of developers builds on systems like ours as well as sister systems in the cloud and sort of begins to unlock that potential across enterprise and public and social good applications. It's going to be exciting.

Okay, excellent. So yeah, definitely going for one up in the tally there. It's going to change healthcare, the environment and science and everything. All right?

I mean, but it is, it is, I think, I think there is this huge opportunity to unlock that, that utopia that Natalia was mentioning. And I think we are optimists, but we're also pragmatists, right? We, we know that in order to achieve that, that optimal outcome, we need to give people the right tool, we need to build the right tools so that we can not just build the exciting new applications, but we can also figure out how to do that in a responsible and safe way. I know we didn't talk a lot about that today, but you don't just need high performance compute to build the right model for your application. You also need a high performance computing platform to research different data set mixtures and different techniques to make sure these models are not just accurate but they're also responsible and safe and built on an energy efficient platform. So all these factors come together. I know we don't have time in just today's conversation, but I think building the right infrastructure underneath all this is a really important part of not just unlocking those new methods and applications but also doing that in a way that preserves safety and responsibility and allows folks to do this in a cost efficient and energy efficient way.

All right, nice. So both very utopian visions. Yeah. Very excited about this. There's lots of good stuff to come and I think that's where we more or less started the show. So I think that's where we'll end it as well. Thank you so much, Natalia. Thank you so much. Andy. It was great chatting to you both.

Thank you. Really appreciate it.