Data Exchange Podcast (Episode 185): Andrew Feldman of Cerebras

All right, so today we have Andrew Feldman, CEO and founder of Cerebras, who I met many years ago. And in fact, I somehow talked him into coming to a laundromat in the Mission District in San Francisco. And now, to my defense, we weren't there to do my laundry. It was actually a laundromat slash coffee shop, which no longer exists. So Andrew, welcome to the Data Exchange Podcast.

Hey Ben, thank you so much for having me.

And, um, all right, so for those of our listeners who are not familiar with Cerebras, Cerebras was part of this cohort of, uh, specialized hardware startups focused on deep learning. That came, that started several years ago. But Cerebras is one of the thriving survivors, I would say. And in fact, one of the reasons we're doing this podcast is they've made so much progress that they're now giving away open-source models, large language models. So Andrew, so what— first of all, Cerebras GPT, maybe briefly describe what motivated this project and are there any updates?

Sure, Ben. As you know, at Cerebras, we build some of the fastest AI accelerators, and in November we announced that we had built a cluster, a supercomputer with, you know, more than 13.5 million cores. It was built up of 16 of our machines, and this was one of the largest AI supercomputers built. And immediately customers began asking to use it, and we began renting it out. We saw an opportunity to train on this cluster an entire class of GPT models, not one, and put it in the open source community, but 7. And to train with sort of top-of-tree expertise, train it to the Chinchilla mark, to use MewP, to use all these things that had been sort of accumulated over the past 6 or 8 months, of, of research in progress. Uh, and so we, we trained all these models. We did a, a scaling law study so you know exactly how to predict the amount of compute necessary to achieve a desired level of accuracy. And then we put it in the open source community, and we, we did so with an extremely permissive license. We use the Apache 2.0 license, so, uh, you are encouraged to, to to take the model, to use it, to train, to do inference, to use it for profit, to use it for research. And to our, our great surprise and delight, 250,000 different users downloaded it and continued to use it. We posted it on both our GitHub and on Hugging Face. And, and it has been an extraordinary success. I, I think this was, uh, and induced a collection of others to, to put big models in the open source community as well.

And I know, is this like a, a specific snapshot on in time, or is the— are you guys continuing to publish new checkpoints and new updates, or—

that's, that's a good question. We published not just the, the checkpoints, not just the trained parameters, but we also publish we published the training methodology, the recipe. And so everybody's encouraged to take it and run with it. We use the Pile dataset. So we use the open source dataset. And we are planning to provide all sorts of additional capabilities down the road. But right now, with hundreds of thousands of users of it. It's actually a great joy to travel and to meet somebody thousands of miles away, and they say, you know, we're using Cerebras GPT, and here's what we did to it. We put our ideas on top, and we've been fine-tuning with this dataset. And what a really nice feeling.

So are there plans to to a completely new family using a different training dataset and a different model architecture? So, or this is it for now?

No, I think you will see from us a collection of different capabilities put into the open source community. I think we very much believe that a rising tide lifts all boats. We definitely want to avoid a scenario where 1 or 2 or 3 companies are controlling the training of models. I think that's bad for the ecosystem. I think it's bad for the ecosystem that those who are already large are arguing for regulation and for prohibiting others to get large. I think there are— this is the way a healthy community behaves. It behaves with many people doing work, publishing their work, learning from each other, inducing the type of creativity that we've seen over the last 2 years in AI.

And so have you heard from like other researchers who have actually taken your work and basically extended RAM?

We have.

Yeah. Yeah.

We have. It's great. There's a whole Reddit group and questions are being answered and asked every day. We, we've seen people extend it. We've seen people use it to train other languages. We've seen, uh, uh, a really, uh, sort of interesting and exciting, uh, proliferation of, of use cases and applications.

So, uh, we've, we've talked about this in the past, but I, I think we're both in the camp that, uh, As far as enterprises will probably end up training kind of custom foundation and language models for a variety of reasons, right? So, uh, I do believe that, yeah, security, uh, IP, and the desire to move faster than maybe the API provider. Um, so what has been the reaction from companies? Are they— did they, uh, end up approaching you and say, hey, yeah, So this is doable. How much would it cost to do one from scratch, or can you help us fine-tune these things?

Then that's exactly what we're seeing, and we're seeing that around the world where customers, especially larger enterprises, don't wish to be dependent. They don't wish to, uh, to send their data to some third party, uh, and what they're looking for is not a general, uh, GPT, but something that does, uh, helps them in their line of work based on their data. And so they're frequently training models, uh, under 20 billion, somewhere between 6 and 20 billion in size. They are, uh, using their own data assets and looking to leverage those. They are sort of thinking about ways to convert their internal proprietary data into, uh, into something to build on. And I, I think they're customizing GPTs. They are, uh, and we're seeing that the same request in, in Singapore, in Switzerland, in the Middle East. We're seeing very, very similar patterns of requests.

So, so sometimes I get the sense that a lot of the conversation and a lot of the articles are focused on training and fine-tuning. But then, you know, once these things are built, you start realizing, hey, a couple of things. One, they're actually expensive to operate, right? So the inference can get expensive. And actually, I don't think there, there's been a lot of work on optimizing. So in other words, if you start using these things through an API, you know, before when you're doing NLP using a library, you're just used to like fast, fast answers, right? So now you've got to wait, send the thing over. Now you're going to wait like 15 seconds for an answer, right? So what should we expect in terms of progress in terms of deployment, inference, and all these issues pertaining to actually operating these things.

Ben, I think it's a really important point and one that doesn't get enough attention. I think it's autoregressive models, right? Like these Big Chat models are extremely expensive to do inference on. And I mean, the truth is they're so expensive that OpenAI and Microsoft started capping your use, right? They wanted to do it for free. They were too expensive even for, for Microsoft. And so, um, uh, we, uh, are seeing, uh, especially for those who are trying to give it away for free, uh, an effort to stay under 7 billion parameters. The cost of inference scales linearly with parameter count. right? So a, a, a 14 billion parameter model costs in compute about twice as much as a 7 billion parameter model. Uh, so we're seeing a couple trends. We're seeing one, people working on smaller models. Two, uh, and this, this started sort of with Llama, uh, they're showing smaller models more data, right? And, and so the amount of cost to train is pretty simple. It's a function of approximately, uh, the number of parameters, times the number of tokens you show to train. I mean, that's a reasonable rule of thumb for how expensive it is to train and compute. And so you can choose to spend that budget with more tokens for training or with larger models. And what we're seeing is a preference for smaller models and more data. And in part, that's because it's a recognition of how expensive inference is down the road, that they're willing to pay more today to train smaller models to drive down the cost of inference tomorrow. And that's a, a very interesting thing and something that probably wasn't predicted by anybody 2 or 3 years ago.

And actually, for those of us who, who use these things, it's interesting. You start developing tricks for how to use these things. Like, if you want to do a certain thing, you go to a big model, But if you want to do a lot of small things that maybe the small models are good enough, you go there because it's cheaper, right?

So I think there'll be a lot of that. And I think often with the new technology, there's a one-size-fits-all starting point.

And or the phrase is AGI. People think there's only going to be one, right?

So yeah, I'm a little less I'm, I'm not so much a believer in that, but I, I think we do have a pattern in the development of technology where at the beginning it's a one-size-fits-all, it's a brute force approach, and then people realize it can be done, and then they look around and say, oh wow, now it's really expensive. And so the first thing they do is to try and think about how to get out of the one-size-fits-all and, and allocate small problems to smaller compute blocks and big problems to bigger compute blocks. And right away that saves money. And that's exactly what you were describing.

And in terms of the economics for enterprises, it seems like a starting point might probably be take one of these open source models and work with someone to help you fine-tune it, right? So, and there's a bunch of startups coming out that are developing tools for tuning these models. But do you think, Andrew, we'll get to a point where training from scratch for an enterprise becomes a realistic option, or are models always going to be in the several billion range, in which case it's not economical to train from scratch?

You know, we, we're finding for large enterprises, many of them want to train from scratch. I, I think if they have enough data, a trained-from-scratch model will outperform a fine-tuned model. And so I think they're wanting to train from scratch. I think, will they train from scratch on models 50 billion and above? My guess is probably not. But I think they can get really extraordinary results by having well-curated, high-quality data and delivering that on smaller models. I mean, one of the reasons we need to go so large—

Define small.

Um, 7 to 20, 2.7 to 20, something in that category. Um, I, I think one of the reasons that we need so many tokens to train these big GPTs is so much of the data is shit, right? That this is internet scraping, that this is filled with garbage, and One of the things that's really interesting when you begin to work with enterprises is their, their datasets aren't a trillion tokens. They might be 100 billion tokens or 50 billion tokens, but they're high quality and they've been cleaned and there's not garbage in there. And that gives you very, very high caliber results. And so you can use fewer tokens per parameter to get a good result when your data's extraordinary. And, and that's true in, in, All statistics, right? If, if your data's less good, you need more of it. And, and so there's no surprises there whatsoever. High quality data on smaller models, either trained from scratch or fine-tuned, is producing really good results for major enterprises around the world.

So without revealing the exact detail, exact figures, So what would— for our listeners, what's like a ballpark for a 2 billion parameter model trained from scratch? What cost are we talking about here?

You know what, it varies based on how many parameters you want, but we just finished training for a customer a 13 billion parameter model on a very well-curated 50 billion parameter dataset. So that's a small dataset. For that size model. On the other hand, it was extremely thoughtfully curated. It took a couple days to train on our cluster, and it's producing exceptional results. And so I think we're seeing that across the board. I mean, what does— if you just step back and you think about what, you know, what do some of these Foundation model companies not have? I mean, they were started in the last few years. They don't have any data. So their data all comes from scraping and from borrowing and from third-party images and all this stuff. What, what do large pharma companies have? They have huge, interesting datasets. What, what do, uh, large energy companies have? They have extraordinarily interesting datasets curated over decades that they're looking to find insight in. And so while these datasets might not be as big as, as, uh, sort of the, the, the, the open source scrapings, they're vastly more appropriate for the specific work of each of these companies. And so they're using, um, better data, cleaner data, uh, data that's, uh, um, uh, deduped, um, and is directly relevant to their problem.

So in the example you alluded to, you kind of described it as, you know, it took about 2 days. So I interpret that is that after 2 days you get a model that's really useful already.

It was well trained. The model had achieved accuracy. The loss was asymptoting. And so now might we have gotten better results with more data? We might have, but the cost of improving accuracy had begun to increase dramatically. Right, right. That's generally where you decide your model's well trained.

So 2 days of training, it seems to me that's something. So do you expect some companies who are faced with this— let's assume they can afford it— do you think they would do that every quarter or a year?

I think it depends on the rate at which they're accumulating more data. I, I think if they're, uh, accumulating valuable data at a meaningful rate, that they will want to do that as they get enough new data to, to move the dial. I, I think many of these companies, uh, have, uh, big piles of data, and they're gathering it at tremendous rates. Um, they're licensing it, they are generating it, um, and I, I think a lot of their enormous research and development budget, uh, is captured. Its work product is this data, right? And, and that is something they're spending on every day, and it's captured in this data, and they want the most accurate models they can. So I think they're going to train it frequently.

So, so, um, as best I can tell, so the, the, the best models out there that are proprietary are, uh, GPT-4, PaLM-2, and whatever the name of Anthropic's model is, right? So, and if you look at some of the rankings from UC Berkeley, the LM-SYS, which where they use Elo from chess, it seems like the proprietary models are still quite better. But granted, this is general purpose, like you said, right? So it's a different ballgame when we're doing domain-specific things. But, uh, with that said, um, you know, so the gap seems still, uh, big. So the best, the best open source model in that ranking is still maybe 200 points behind. If you interpret that in terms of this Elo ranking, that means head-to-head with GPT-4 versus Vicuna, GPT-4 wins 80 82% of the time still.

It does, but it, you know, the, the, the, the, the Vicuna llama, uh, or whatever the Vicuna offering is, didn't cost, uh, billions to train, right?

Right, right, right.

It didn't take years and cost billions to train. And also the open source is still getting started, and the open source is, uh, it, it, it is just getting started. I think you're exactly right. And so I, I think a couple things. I think first The work being done by OpenAI and Anthropic and Cohere and the guys in Germany, Aleph Alpha, these are really interesting, interesting work. And I think there's great value in those models. I think they are all consistently outperformed in specific tasks. By proprietary data on general models, on open source models for specific tasks. And they continually outperform open source models on general tasks, right? And so it's not at all surprising that you can build a BioGPT based on or fine-tuned on either from scratch or fine-tuned on top of an open-source GPT model that is better at biology-related literature than a general-purpose model. This is not at all surprising.

And this is why, as I mentioned earlier, you'll end up probably using multiple things, right? So my guess is, yeah, if you're doing kind of biology-focused things, you'll do one of these domain-specific things.

Absolutely.

But then when you get something out of that, maybe for summarization, you go to the general purpose.

I think there'll be a, a workflow, no different than, you know, we, we don't store everything in, in the biggest of databases. We, we have some caching, we, we, we have various levels of storage, and then you go into your, uh, your cold store versus your hot store. I mean, if you think about it, that's, that's the way most of our work is done, where we have a collection of different technologies and we, we map the, the workload to the technology based on the need.

So, so, uh, on the model side, so, so the GPT-4 seems to be— and I think Bard, I'm not sure, I'm not sure about PaLM-2, but I think GPT-4 has some notion of multimodality in the sense that maybe you can even upload images to it, right? So the open source models obviously have to catch up in that area. And the other thing that I sometimes worry about, Andrew, is, you know, right now everything's about transformers, but surely transformers is not going to be the end of the story, right? So there might be some new development in a year or two or three. But now we've entered an era where the disclosure is much less. It's in the technical reports. There's hardly anything in there for, for GPT-4 and PaLM-2 in terms of the model, right? So GPT-4 tells you they're using transformers. But down the road, what if there's the new transformers and they don't even tell you? They just tell you we're using something else, not transformers, but we're not going to tell you how it works, right? So.

I, I think the, the dynamics of how that will unfold is still to be written, right? I, I think, um, uh, you know, changing off transformers is, is interesting. It would cause the GPUs to be vastly slower because they have specialized circuitry now.

Yeah, yeah, I'm just speculating that there's some development on the side. That, that will happen which we may not know about.

I, I think for sure, yeah, that, uh, that the proprietary teams who are working on these problems have special sauce that they're adding on. And the question is, is whether that— and this is sort of the competitive question whenever there's proprietary versus—

there's a fundamental break, just like Transformers, right? Fundamental break, right?

So That's right. And that's the question is whether their special sauce can outweigh a community working against them, right? That's the question. Is there enough value in their special sauce to overcome the fact that thousands of other engineers are working on it for free?

So I guess the thought experiment is what if Google had transformers and never talked about it?

Um, I, I think it, it is an interesting question whether it's interesting. I mean, it's it, and I think it's really interesting in light of some of the, the, the leaked memo they wrote saying, look, uh, we, we don't have a durable advantage. OpenAI doesn't have a durable advantage.

Yeah, that was just one guy though.

That was one guy. That was one guy. I think it's, it's very interesting. I think that's probably truer than anybody would like to admit that they don't. I think they're— everybody thought Google's vantage was sort of impossible to overcome and boom, disappeared.

Yeah, yeah, yeah, yeah. So we've talked about models to some extent, to some extent data. Of course, the big element is compute, which is where you guys play in. I was just talking to, uh, some people who are well connected in China a few days ago, and I asked them, you know, with, in light of this kind of trade thing, what's the state of LLMs in China? And basically the high-level explanation is, uh, you know, if it's GPT-3.5, we have the hardware, but, you know, beyond that, We don't have the hardware. So, so how much does hardware matter in, uh, in this world of if you want to be at the most cutting edge?

Hardware is the defining part of cutting edge. Um, and I, I think the government saw that when, uh, they asked NVIDIA to, to stop aiding the Chinese building an AI capability. Um, I think It is fundamental. It is why OpenAI has required so much money. It is sort of at the ground level. The problem of AI is a problem of putting to work an enormous amount of compute. And that problem will not go away. It has grown at an extraordinary rate. Both, both sort of the amount of compute needed, uh, the amount of data used, um, these are, are continuing to grow.

Um, so what has been, uh, so for our listeners who are not familiar with Cerebra, so what has been kind of the, uh, uh, progress on the Cerebra side of hardware in terms of, uh, Sure.

I think when we started the company, our first part was the largest and fastest part on Earth that had 400,000 cores. It was vastly faster than clusters of GPUs. We've more than doubled. We now have 850,000 cores. Customers are now not buying one, but they're buying clusters of 16 or 32. I mean, it is they're putting to work tens of millions of cores on a single problem. And that is the size of some of the largest supercomputers on Earth. So the largest supercomputer on Earth has about 8 million cores. Now those are bigger, stronger cores, but still trying to coordinate more than 25 million cores to work on a single problem is an enormous challenge. And so what we've seen is an exponential increase in the amount of compute customers want to work on these problems. They want—

What about memory? How have you guys evolved in terms of amount of memory?

You know, we have a very innovative approach. We have a huge amount of memory on chip. We have 40 gigabytes of memory on chip. And so this is thousands of times more than a GPU. We have tens of thousands times more memory bandwidth, which is the fundamental limiter in most GPU work. But also our approach is to, to— we invented a technology that allows us to store parameters off-chip, so we can support 1, 10, 100 trillion parameter models even on a single system. And that's how many, how many? 100 trillion on one, on a single CS-2 machine.

So, but it, uh As far as the user, it kind of feels like memory.

It is memory.

It is memory.

It is memory. Yeah. It is storing your parameters.

And so this is also useful for inference, obviously, right?

Also useful for inference. Yeah. I think, as we mentioned earlier, In the, in the old world of, uh, CV inference and, uh, sort of simple NLP inference, uh, you could always fit it on a, on a small processor, a CPU or a little GPU. Um, now that we're doing big autoregressive inference, people are using clusters of GPUs. They're using, uh, one of our machines that had traditionally been for training they're using for inference. Um, it's a much, much harder problem in this big autoregressive world.

So, so the Andrew— the A100 also has about 40, 80 gigabit— gigabytes of memory, right?

No, that's memory off-chip. Uh, the memory on-chip is, is absurdly small. Memory on-chip is, uh, the, the A100 has— came in two configurations.

So for our listeners, uh, What's the, so what's the difference? Why, why is the dis— why the distinction between on and off? Yeah.

Look, I, I think, uh, imagine you're watching a ball game, uh, you, you're maybe watching some football and, uh, you have beer in your fridge. Uh, that, that's like having, uh, memory on chip. It's nearby. You can go to it often. You can get a snack. Now, memory off-chip is like having beer at Safeway, right? To maybe at halftime you can get out of your car, get out of your house, drive to it, buy it, stand in line, drive back, and maybe all your neighbors are doing the same thing. And that's like off-chip memory. And the size of the road and the queues at Safeway are memory bandwidth.

Right.

And, and so, uh, you can access on-chip memory thousands of times faster. You use a tiny fraction of the power to get there. And in our architecture, there's no contention. Whereas in a GPU, by putting the memory off-chip, all your cores are trying to get to it at the same time. And for anybody who's trained big models, this is a fundamental challenge. Is you will almost always run into memory and memory bandwidth limitations.

Right, right, right, right. Um, so are we— are you anticipating models will keep going bigger and bigger? Is that kind of— is that the assumption of the company?

No, I, I think, uh, we are, uh, point of diminishing returns. Well, I, I think we're at a point where people are thinking about approaches that aren't brute force.

Yeah, yeah.

And also the operational complexity and the challenges of actually working with a 500 billion parameter model or a trillion parameter model is very hard. Yeah, right. You, you want to debug a layer of a model that big, it's a beast. Um, you want to think about how to distribute that in traditional approaches over thousands of GPUs. I mean, one of the things that's interesting is if you look at the Llama paper, you look at the, the GPT-4 paper, in the back they have 100 people who are involved in distributing work.

Yeah, yeah, yeah, yeah. Right. Who can afford that?

Who can afford that?

Um, it's basically distributed systems, not really modeling anymore.

That's right. What's happened in that is what you see is the problem of distributed computing has overwhelmed in complexity the problem of building an AI model. And I, I think there are very few companies who can afford to make that trade-off. And so, certainly in the open source community and in large enterprises, what we're seeing is a focus on smaller models and better data, more and better data.

Yeah, yeah, yeah. And so, so obviously in computer vision, they don't use as many transformers But they had, they have years and years of honing tricks like compression, quantization, distillation, because they have to deploy these things in small cameras, on sensors, and things like that. So, uh, is there equivalent effort in LLMs?

I think it's starting. I think we're seeing tremendous effort in quantization. We're seeing people try and do inference in 4-bit. Uh, we're, we're seeing, uh, people experiment with very sparse inference. We're thinking, we're seeing, and we're working on both of those. I think people want to do inference at the edge. And I think we saw that in the amount of space in a cell phone chip. That the inference engine is now taking up, right? There's an estimate that, that on, on an Apple cell phone, the chip, the primary processor there, about 15% of the die area is taken up for your inference engine. Um, and what they want, of course, is the biggest model, the, uh, most accurate model compressed down to work on the smallest chip for the least power. And so there's a tremendous amount of of effort going into that. I think there's also effort in that smaller models could run on, on not just on GPUs and on our machines on custom, uh, hardware, but they can also run on CPUs.

Yeah.

And so the CPU vendors are extremely interested in these techniques that allow models at a billion 3, 2, 7, 6, 7, 13 to be able to claw back into the range where they can fit on a CPU. So there's a lot of interesting work being done there. And in fact, you saw that right away, both with our, uh, with the Cerebras GPT as well as with the Llama, where what do people wanna do? They want to run 'em on their laptop.

Right.

Right. That's the, the first thing people were doing was trying to figure out how to run it on a, uh, on their Mac or on their laptop. And that, that's super interesting. And I, I think, uh, it, it's really a direction for, for the future.

Yeah, yeah. I mean, I think, uh, as people become used to interacting with these systems, they want to be able to use them offline. That's right, that's right. Yeah, yeah. And so I think a lot of these things are going to be must-haves. By the way, so there's also the software stack that accompanies hardware, right? So, uh, famously, uh, NVIDIA has CUDA. But there has been kind of efforts to kind of develop open source alternatives, most recently around PyTorch 2.0 and Triton. But I don't know, I mean, it seems like those initiatives are somewhat stalled or slowed down. But so what about you folks? So how is your software stack progressing? I guess it only benefits your hardware, right?

Well, yeah, I think CUDA is a low-level programming language. I think you're hard-pressed to find anybody who likes writing in CUDA. I think there are a few in universities or people who like to write underneath C.

But a lot of the frameworks.

That's right. That's right. The frameworks were invented because people didn't like writing in CUDA, right? That, that's why we have frameworks, is people didn't want to write in low-level, uh, code. They want to write in Python. And I, I think, uh, when you look at it, about 90% of the market now is, is running PyTorch, um, for, for language, for vision. It's a little different. Yeah, yeah, for language. And, um, I think PyTorch 2.0 is a, is a huge step forward. I think the delivery of MLIR, you know, 18 months back gave the community an, an open source intermediate representation to target. I think the recognition among first Facebook and then by spinning out PyTorch, the PyTorch community, what was that more attention needed to be paid to the fact that this was gonna be lowered not just to, uh, to GPUs but to other hardware that included TPUs and, and the other hardware made by, uh, by Amazon and other companies including us. I, I think all of that benefited the community so that now companies like us and others can take your, your PyTorch and run it on our machine. You can run it on a CPU, you can run it on a GPU, you can run it on, on a TPU. And we, we all can, can run it about equally as well.

Well, supposedly AMD has good GPUs, right?

So AMD, well, I, I think so.

So what's the, what's the missing ingredient there? It's software, right?

Yeah, I, I think, look, um, uh, I, my last company was acquired by, by AMD and, and they, they were in a pickle in, in 2012 and, and Lisa to her tremendous credit, Mark Papermaster and the team there turned around the company. They, they built their CPU franchise back. They, uh, took huge share, um, and that was their focus. And I, I think, uh, they kept the GPU, uh, targeted at gaming. And maybe last year or the year before, they began— they lifted their head up and said, our CPU house is in order now. How do we attack the GPU house? And, and that takes years too. And I think they have an excellent GPU for gaming and they had some work to do to point it at the AI software problem. And I think they will get it too. I mean, I don't bet against Lisa and Mark.

So, you know, as these open source models get better and tools for fine-tuning models become more accessible, I guess for companies, Andrew Feldman, it will— so their crown jewel will be their data. So the moat is the data.

Data is the new gold.

Yeah, for sure.

The, the moat is the data. I, I think if you have interesting data, uh, you will be able to use it, um, either from scratch or in fine-tuning. You'll be able to use it, uh, on your proprietary models or on open-source models. I think It is the data that encodes the insight. And the, the AI is getting access to the, to the insight.

So then, so then the, I think the, so, so there's, it's one thing. So once you fine-tune a model, it's one thing to make it accessible internally, either, you know, your developers, data science teams, data teams can use it through an API. Or even your internal knowledge workers who will interact with it through a chat interface. But I think, uh, it's unclear to me, uh, if, if company— how soon before companies will feel comfortable just, okay, customer-facing, we trained it in our data, now I'm gonna put it out there so customers can talk to our thing, you know. I mean, well, and, and we know that all these things can be jailbroken and, you know, can misbehave, right? So—

Look, I think we sometimes forget that, you know, let's take support as an example. Right. And it's probably no different than other knowledge worker, call it wealth management advisors. They're wrong too sometimes. We don't call it a hallucination. When the support guy tells you a debug path and it's dead wrong, We don't publish articles about it. They're wrong all the time, all the time. And that's why customer support is so challenging.

But they're on the phone, unless you record them, you have no proof.

That's exactly right. I think you're exactly right. They're wrong all the time.

Yeah.

And I think, you know, with somebody who works in this space, I have a different view. And I see this all the time with self-driving cars. I think, Uh, uh, you know, we, we ask of new technology an impossible task. Um, humans are terrible drivers, right? We, we kill in the US, what is it, 40,000 people a year. It's the number one cause of death for people of under 40 or whatever. Um, I, I don't need self-driving cars to never kill anybody. I just want them to kill fewer people than people kill when they drive, right? I, I would like my AI to be wrong less than the support person. I would like it to be wrong less than the wealth manager or the lawyer or whoever it's replacing or augmenting. I would like it to catch errors from the radiologist. I don't need it to replace the radiologist right now. I'm not ready for that. But I would very much like it to catch errors. I would like it to produce an alarm if the pilot is behaving erratically. I would like— right? I would like to bring it in where humans are frequently failing.

But it seems to me there's lots of opportunities here for entrepreneurs wanting to build kind of some of these tools around providing some of these guardrails, the ability for the system to learn from human feedback.

Of course. Ben, I was talking to a startup today, an extraordinarily well-funded startup, and Andrew Feldman, they train their own natural language models. They produce extraordinary results, but like everybody, sometimes they're wrong. And so what they do is they run another model on top of the results. And there they do several things to try and ascertain whether the results produced by their model is garbage. And what they're able to do is narrow down and eliminate almost all the hallucinations. And that's very interesting. And, you know, first we were so happy that we invented models that could get it right 60%, then it went to 70% or 80%, but they're still wrong sometimes. And now we're going to invent other techniques to double-check our models. And step by step, we're going to ring, ring out of the system their propensity to make mistakes.

So it seems like because of so much developments in LLMs, I mean, every day there's something new.

It's amazing, isn't it?

Yeah. Somehow we've, we've— I feel like somehow computer vision has been kind of not forgotten, but, you know, there's not as much discussion around it, right? So the reason I'm bringing it up is I know CVPR is next month. No one is even talking about it, right? But I may— maybe it's just a question. Maybe it's because, you know, obviously LLMs are text language, so people— it's a great interface, right? So for people, but I think that there's equally important and interesting things happening in the vision space and speech.

Tremendous. Tremendously important in the vision space, in the speech space. I think there's a sort of upper bound to the amount of noise that a given industry can make. And I think ChatGPT sucked all the air out And that cool stuff in understanding radiologic— radiological images, or cool stuff in understanding 3-dimensional models with AI tools, really interesting things in reducing the cost of 100 elements of the way we work, live, and play in vision, they sort of got subsumed by the fact that with your phone you could ask questions of natural language processing models and get extraordinary results. And they sort of— the, the interface. And remember, ChatGPT was not a technology invention, right? It was a user interface invention that sat up on top of, of 3.5, right? Which was— or 3.0, which was the real innovation and that nobody used, right? And so I think we had a little bit of that with the Stable Diffusion moment where everybody saw the power of—

And then that's still, a lot of people are still—

It's huge.

I still use that stuff.

I use Stable Diffusion. I think I also use Midjourney. Midjourney. Midjourney has never raised any funding, but man, they're, I think if you look at, uh, some of the movies created by Runway, I think that this is extraordinary. It's extraordinary time. And I, I think, uh, I think if we're at the same spot with, with sort of language in 2 years, I'd be really disappointed. I think you're going to see, uh, the integration between the two. Uh, I, I think you're gonna find what we call deepfakes today sort of profoundly impossible to separate from reality. I, I think you're gonna be able to, uh, uh, vision and text and speech are, are gonna mix in very, very interesting ways.

Yeah. Um, yeah, yeah, yeah. And with that, uh, thank you, Andrew Feldman.

Hey Ben, it's always fun to talk. It's good to chat with you, my friend.

Be well.