No GPU Before PMF // Stanislas Polu // AI in Production Conference Lightning Talk

Thank you. Next up, are you here, Stanislas?

Yes, I am.

Okay, good to see you. How are you, man?

I'm doing good.

Okay, so Stanislas, I mean, you'll introduce yourself, and what I can tell folks is you've spent some time at Stripe and at OpenAI, and now you're going to tell us a little bit about how founders and early stage folks should be spending their attention a little bit, right, when they're just starting out. So I'm sure a lot of people in the audience can benefit from this. The floor is yours. Your screen is right here, and I'll see you in 10 minutes.

Thank you very much. Hello everyone, I'm Stan. I'm the co-founder of Dust. We basically build a platform that brings humans the company data and models. The goal is to let every employee create assistants to supercharge their, their work. That covers from incident response assistants for engineers, code assistants for engineers, assistants that create, prepare your meeting notes for product managers, etc., etc. I started working on this 18 months ago, and before that I was a researcher at OpenAI for 3 years where I worked on studying the reasoning capabilities of large language models. And in particular the mathematical reasoning capabilities of large language models, and in particular in the context of formal mathematics. So I have the kind of unique position of having trained thousands of models using probably millions of GPU hours over my 3 years of work as a researcher, and at the same time being part of the community of people building product in the space for the past soon-to-be 2 years. So I think I wanted to share some thinking there about how the ecosystem is shaping and how the technology is shaping. Maybe not some answers, but at least some directions for founders and people building in the space to think about where the technology is going and how it should impact the way they think about products and their infrastructure. So I think the one interesting thing that has been a very differentiating force between different startups operating in the AI space is B2C and B2B. It's very interesting to see that we are definitely in the B2B bucket ourselves, so really try to sell to companies. But when you think about all the companies that are doing B2C stuff at this when you're doing B2C, obviously you have large volumes. And so generally the order of priority for B2C companies is currently being cost, speed, and performance. Basically, if you think about Perplexity, you have millions of people asking most of the time very trivial questions. So you want something that doesn't cost too much because you're serving all those millions of people. You want something that is very snappy because that's the, the thing that makes the product sticky for especially in B2C setup. And then last only, you're thinking about performance. I think for the B2B use case like us, we have a completely inverted order of priority because we're capable of charging more. We really think first about performance of models, speed second, and cost third. And I think I'll come back on the cost questions, but basically, uh, when you are in a B2B setup, that's really a setup in which you can probably really just not think about cost today. And it's fine to do crazy things that cost a lot to— for even simple product interactions, because as I'll make the arguments a little bit later, there's a good reason to believe that that cost will diminish very drastically. And so if you are going in the B2B motion, it's fine to absorb that cost during that transition period. The second point that I wanted to touch base is about fine-tuning, because it's really interesting how everybody talks about fine-tuning and everybody wants to, especially when you talk to enterprise, everybody wants to have their own fine-tuned models on their own data. I think it's a little bit of a fallacy that is important to unpack and to think about. Basically, first, fine-tuning of large language models is something that scientifically speaking is not super well understood yet. One very simple example of that is that we still don't know how to do online training. That's how bad we are at fine-tuning. We've created an artifact through pre-training that is called large language models, and we have a new feed of data coming, which is the feed of data coming from the world, And we don't know today how to incrementally update those models. So we're really bad at fine-tuning. There are things that we do well, like alignment and stuff like that, but the science of fine-tuning is really not well understood. And so it's important to understand that. And second, fine-tuning on company data, as an example, is something that is meant to not work. It's something that many people are offering to many companies because that looks cool to have your model has been fine-tuned on your data. But for use cases that are varied and that are generally the ones that happen when people interact with models for productivity reasons, that fine-tuning is very likely to not work. Let me explain why. Fine-tuning really— I mean, if the tasks that you're interested in are very varied and quite informal, then only the largest models will really be good at that. If you take a small model and you try to ask very varied questions about complex stuff, you're generally going to have much worse response than with a large model. And so you're stuck with large models for those kind of questions. And now, if you try to fine-tune a large model, generally the, the dataset that you have access to, uh, let's say the— all the data that has been generated by a company is really too small to effectively fine-tune a large model. And so the reason why is because the model is so gigantic that you won't be able to move the internal knowledge graph of that model sufficiently. And so what you're only going to be training the model to is to, given the knowledge graph, to invent more data that looks like the company data. And so you're bound to have a lot of hallucination. And so for those reasons—

just quick question, just in case, are we supposed to see a screen other than what we're seeing? It shows no PMF.

No, no, no, I just had one slide, so that's fine. Good.

I just wanted to make sure.

You can even remove it.

I'm happy to show my face.

Full screen. Um, and so, um, I think the, um, one motto that we have at DUST is no GPU before PMF. And I think it's important to realize that trying to train your models as a startup might be your core strategy, and there's a lot of startups that are really proficient like that. We can think about Mistral as an example, But for many startups out there, it's probably the wrong strategy. It's very fashionable to train your own model. It's very fashionable to use GPUs. It's very fashionable to create a moat around training. But the reality is that you're just creating a small rock that might be washed out by the next generation of models. And so it's very important to think hard about whether you want to open that box. Really, startups are not meant to do research, and fine-tuning still is something that is within the research realm as of today. Um, to kind of give some usable stuff for the audience, I think it's interesting also to think about what's— how the ecosystem will shape in the future. It's very interesting to, to realize that GPT-4 is soon to be 2 years old, meaning that the first version, the first end of training of GPT-4 was almost 2 years old. Given the space in which we are, that seems like an eternity. So for those past 2 years, that model has been the best model and most of the competition has been catching up with it. But it's interesting to also realize that OpenAI hasn't released a new model since then. And it's interesting to ask yourself the question as to why. One thing is that scaling those models at the scale necessary to go to the next scale after GPT-4 is extremely difficult. And the second is that the scaling laws that govern the performance of models as you increase the compute are scaling laws— so that's basically power laws— with a very steep exponent, meaning that as you add more compute and more data, the amount of compute that needs to be added at the end to really make a difference becomes gigantic. And so basically we have two alternatives in front of us. Basically in the past two years we've had GPT-4 as the king model that is much better than any other model, and all of the open source and other players have been playing catch-up to GPT-4. We are all today entering a world where we have serious competition to GPT-4 Gemini Pro 1.5 and Gemini Ultra are serious competition to GPT-4. And even the open source community is catching up with GPT-4. I think it's legitimate to think that Mistral is capable to have a model that is at GPT-4 level within the year. And so we really have two interesting alternatives in the future. It's whether GPT-5 or the next generation of model will be substantially better than GPT-4, or not. And if you think about the scaling laws and the fact that GPT-4 has been there for 2 years, there's kind of a high probability or an interesting probability that the next model might not be, from a human perspective, might not be substantially different than GPT-4. And so if that's that scenario, that's going to be very interesting because all the competition will catch up with OpenAI and we'll live in a world of highly competitive token providers and the cost of token will just go to the ground very rapidly. And then the second alternative is GPT-5 becoming, I mean, being released, or the next model being released, whether it comes from OpenAI or Google or any other actor. But the next big model being released is substantially better than GPT-4, in which case we're probably going to end up in a dynamic that is similar to the past 2 years where we have one leader that is capable of making a somewhat high price of token and all the competition trying to play catch-up. Given the fact that GPT-4 is 2 years old, I think my conviction goes more and more towards we might not have a massive improvement of models by scaling. And another signal towards that is the fact that most of the differentiation today comes from going to multimodal, going to longer context, and not going to a bigger model. And so that's going to be a very interesting ecosystem that we're going to live in as all that compute should catch up. And I think it's something you have to take into account when you think about what's going to be the good model to use and what's the strategy around training your own model, etc. So I hope that's been useful to get some direction in that space and some ideas on where the space might go. And thank you very much.

Thank you very much, Stanislas Polu. Please stay in the chat. I would love to ask you questions for about a full day. I would like to have you on. This is fascinating stuff. I mean, I'm just like, I'm teeming with questions. I also think that as a slogan, so many things should not happen before product market fit. Most people that I talk to in like startup world, they're like, I'm trying to do that thing. Just don't do any of that. Just product market fit first. 100%. But the dynamics that we're likely to see and how we should be reacting to the scaling laws, right? And how those scaling laws are already going to impact the strategy that startups should already be taking into consideration. It's fascinating. Stan, thank you very much.

Thank you very much for all of your insights. I'll stick around if there's any questions.