dotAI 2024 - Stanislas Polu - LLMs reasoning and agentic capabilities over time

Phase transition, or phase change, is a physical process of transitioning between one state of a medium to another. It's typically used for the basic state of matter: liquid, gas, solid. When we think about society, we could think about the basic state of society. As an example, the pre-agricultural society to the post-agricultural society. The pre-electricity society to the post-electricity society. We haven't been lucky to live in those past phase transitions, but we are lucky to live in a new phase transition, which is the transition of the pre-AI to the post-AI society. To me, that phase transition started with AlphaZero. AlphaZero, when you think about it, is a crazy technology. Imagine aliens coming on Earth, right? They would, like, say hi, inquire about our society. Eventually, we would explain to them what chess or Go is. Just by having the rules, they would probably go back to their ship, crunch some numbers for a few hours, and come back to us and beat us forever at chess or Go. AlphaZero is exactly that, a technology that just given the rules creates a machine that can beat us at that game. This is the closest thing we've ever seen to alien technology. That phase transition, and there's 7 years between the two, obviously can be characterized by the emergence of ChatGPT, the emergence of agents, the use of AI at work. More recently, one kind of major event is an AI achieving bronze medal at IMOs. What are IMOs? IMOs are the International Maths Olympiads. These are the kind of brightest students from all over the world coming into a room and competing to solve math problems. It didn't get the gold medal, but it did get bronze medal, which is pretty crazy. And this is the first time an AI system was able to beat some of the best humans at something that we consider being the most reasoning AV task we can think of: math. And we start to see the society reacting to that phase transition. This was just only last week. Two Nobel Prizes were attributed to computer scientists for the first time for achievements they made through AI in physics and biology. So we can think ourselves, what is coming next? And to try to get an idea about that, we're going to go back and do a little bit of a history of deep learning. So I'm going to be describing a number of models. And for each of those models, I'm not going to be giving the number of parameters, of the performance on a benchmark. I'll do something slightly higher level. And I'll give you the characteristic time complexity of the task. And the reliability in terms of 3 buckets: crappy human, average human, best human. So AlexNet is the first achievement of deep learning. This is a convolutional neural network. It was applied to image recognition. So it's 2012, basic image recognition, much better than anything that was built before. Characteristic time complexity: 1-2 seconds. Reliability is still of a crappy human. It gets us to ResNet 2015. Just a slight modification to the architecture lets us go much deeper. We go from mediocre image recognition to good image recognition. Time complexity is still 1-2 seconds, one of the frontal cortex of the human. You can just do one pass, you get just one signal, and the reliability goes to average human. Then 2016, emergence of PPO as an algorithm that is still very much used today, applying reinforcement learning to the CNN architecture. We're starting to see good results in terms of gaming around the Atari games. We achieve human-level Atari playing. The time complexity— interesting here is the time complexity of the task goes from a few seconds to a few minutes, maybe 30 minutes or maybe 20 minutes. Time to play a natural game. And the reliability is of an average human. AlphaGo and AlphaZero, 2017. We achieved master Go, master-level Go play. And here, so, that's interesting to dig a little bit into what AlphaGo is. AlphaGo takes the CNN architecture, which is just capable of doing one pass and has been mostly capable of doing what the frontal cortex can do, and embedded it in a search loop. It's not possible to do that for everything, basically. There is many— there's actually 3 core things that make AlphaGo possible. The first thing is that games like chess and Go have a finite action space. We couldn't have done that with an infinite action space. If we had an infinite action space, the CNN architecture would have been able to really handle that. The second thing is that it's a game with rules, so it's a— an environment that you can simulate. And so you can generate a lot of data and apply RL. RL is, in a sense, just simply a way to extract training data from a simulated environment. And the third thing is linked to the fact that those games are zero-sum games, like symmetric zero-sum games, which means that you can have an AI play against another, you will always get signal because there's always a winner and there's always a loser. So what we see here is that we had a basic technology capable of doing anything that a human can do in 1-2 seconds, and then we embedded it into a search loop, and all of a sudden we were able to achieve better reliability, best human, and a longer time complexity. At the same time, in 2017, the transformer was invented. Time complexity, typical time complexity, let's say 10 minutes. It was applied to translation. The reliability of that system at the time, crappy human, really bad translation. We go to 2012 and GPT-2, which is kind of a few years required to scale that technology to a first level. We achieve with GPT-2 for the first time coherent text generation. If we had applied that to the early version of Transformer at low scale or the RNN, the text would just go away into nonsense. And for the first time, we had kind of a couple paragraphs of coherent text. So Those paragraphs were not mind-blowing. Reliability, crappy human. The time complexity, 30 minutes, the time to write them, let's say. We go to 2020, GPT-3. Scaled GPT-2, 1.5 billion parameters, to GPT-3, 300 billion parameters each. And one thing incredible emerged from that, meta-learning, which means that even if we don't have at all the objective encoded in the training sets. The model at that scale were able to meta-learn, which means just by giving it examples in the context, it was able to understand the task and achieve itself. Its reliability was still of a crappy human-ish, time complexity is still 30 minutes, but we could do pretty much anything with it. 3 years are required to scale that to GPT-4. GPT-4 is basically the exact same technology scaled up and gives us the reliability of an average human. Basically, with ChatGPT-4, you can think that you can pretty much do whatever a human can do, an average human can do, in 30 minutes. It gives us text generation. Because it's better, we also have, and it's been also worked on, we also have tool use and the ability for the model to generate actions, which is very interesting. It moves us to 2024, which is this year, where the same story happens again. We take that basic technology, which generally is capable of achieving text of something like a few minutes, and we embed it into a search loop that gets us to AlphaProof. So what is AlphaProof? AlphaProof is applying LLMs to formal mathematics. What is formal mathematics? Formal mathematics is a system where you have a very powerful type system. In that system, a statement, a theorem is actually a type, and a proof is a piece of code. And because the type system is so powerful, the type system can check the validity of the proof when you give it to it. So it's very interesting to apply to LLMs because it's a code generation objective in which you have perfect verification. I myself at OpenAI worked on it between 19 and 22. And Guillaume Lemp, who's now at Mistral, was doing pretty much the same at Facebook. And we had that kind of realization at the time that it's only by finding a system where you have verification that we would be able to embed the LLM into a search loop to try to achieve better performance. What DeepMind achieved is basically taking that same approach and scaling it to a next level. So what they do is that to get a lot of formal data to train on, they actually train first an auto-formalizer, which is a model that takes informal math statements, the ones that you can find on the internet and in exams and stuff, and turn it into formal versions of them. That translation is not perfect, but that's not a big deal because you have a formal system. So if you can manage to find a proof of a statement, even if the statement is not perfect, it's still a good signal. And then they scaled the training and the application of that search loop to all those statements and reached a scale that we didn't reach at the time, and which was what made them capable of achieving that incredible result, which is an IMO medal, a silver medal. And so it's interesting to see that we see the same story happening twice. Basically, in each case, we have a model, the CNN, The prediction at the time was it can probably do whatever, and Andrew Ng was saying that, it can probably do anything that a human can do in 1-2 seconds. Now we have the LLM. We could make the prediction, and it's roughly true, that it can do pretty much anything that a human can do in 30 minutes. And each time we achieve the average human capability by scaling the technology, and each time we also find a system where it was possible to embed that system in a search loop and push it to much better performance. An interesting question to ask ourselves is, is the LLM the last paradigm that we're going to need? If you look at those two curves, they look the same, so it feels like there should be a third thing at the end, right? And so you've probably all heard of the scaling laws, the promise that if you scale those models, those LLMs, you'll achieve better perplexity. The problem with scaling laws is that having a better perplexity, we don't really know what it looks like as, as humans. And so we are very hopeful that by decreasing the perplexity on the internet, we'll get incredible capabilities out. But what we've seen is that as we scale those things, the time coherence of the model was still the same. And I think it's interesting to ask ourselves a few questions about whether LLMs can be the last paradigm. First is, where can we find long horizon data? We cannot really do reinforcement learning against the wall because reinforcement learning requires a very high sample complexity. And it's just not a simulated environment. And if you look on the internet or wherever, there is no training data for very long time horizon tasks. There is no training data for starting a company. There is no training data for inventing a new theory in mathematics. There is no training data for curing a disease. Another interesting question is why all the labs landed on the same SM TOP. Basically, if you think about the reasoning capabilities of the LLM, we achieved the level of reasoning of GPT-4, was unbeaten since it was trained, but also matched by every lab. Despite the very strong incentive for labs to try to push that technology at a larger scale, Google can push that technology at a much larger scale. OpenAI has what it takes to push that technology at a much larger scale. They have clusters that are way bigger than what was used to train GPT-4. And yet there is really no big difference between the reasoning capabilities of GPT-4 and that of the recent version of GPT or that of every other model that we see out there. And finally, where is generalization? Those models definitely can generalize. They generalize really well locally. But you can think of generalization and reframe it by thinking about transfer between modalities. Basically, it's well documented that there is very little generalization between code— sorry, very little transfer between code and other reasoning tasks. The other very surprising thing is that as we added new modalities, we added an enormous amount of training data, like the audio tokens, the video tokens, the image tokens. And so we would have expected to get much better model by adding those new modalities because the training set was much bigger. And we also would have expected, as an example, that because we train on images, the model should be much better at reasoning about space. And yet that's not really the case. So we're seeing a lack of transfer between those modalities, which is very troubling. We're starting to hear those kind of statements, which is, we're saturating the data. Top AI scientists can say regularly, the LLM feels great, but they also not feel quite right. There might be something else. If we look back at what we covered, at least my prediction is that we're probably on the verge of having a new paradigm. And maybe the LLM is not the last paradigm. If we frame it in this way, I think the next frontier is really building a model that can inherently think. If you think of the CNN again, it can see, it can do anything that it can do in a few seconds. It was a lot of research and then a scaling phase to achieve the human level, basically, of that task. If you think about the LLM, it can generate. Because it can generate and because of the autoregressive sampling nature of it, it can cover tasks that maybe take more, something like 30 minutes. And it was a lot of research and then it was scaled all the way to today. But inherently, those LLMs, they really cannot think. Even if OpenAI is trying to convince you that O-1 is a thinking machine, it's just kind of a way to exploit a little bit more test time compute by generating more tokens. But inherently, the model just in one pass generates the next token. It doesn't have a place where it can just think very hard for a very long time. And that architecture just does not allow it. And it feels like there's probably another architecture that should peer for being the architecture that can think. And there we can imagine the craziness that would be. Maybe we would be able to cover tasks that takes days or even months or years. So that realization is what basically motivated me to leave OpenAI and go to product. At the time, it was September 2022, GPT-4 was trained. It was available internally. And it was at the same time extremely exciting because it was a notch better than GPT-3, but at the same time extremely underwhelming because we had spent months and deployed a gazillion GPUs to train that massive model on a dataset that was never achieved. And the artifact that you get at the end, kind of the same. And yet, The capabilities that that model has, which is, as I described, basically the capability to do anything that an average human can do in, let's say, an hour or 30 minutes, is incredible. It has the power of deeply transforming the society. Even if we don't get yet to general intelligence, the impact that that technology will have is humongous. And yet at the time of September 2022, the technology was completely underdeployed. The revenue of OpenAI was something like in the tens of millions. Pretty much nobody was using that. So this was kind of the core motivation to go from research back to product. It was the motivation of saying it is probably at the product layer that we have locks that we need to unlock to release that usage. Obviously, ChatGPT came through there, which kind of confirmed the hypothesis that the product layer was the missing piece. And when we think about the use of AI at work, which is the real interesting bits, try to work better, try to work faster, we are probably still in a pre-ChatGPT world. The impact, potential impact is humongous, and yet it's underused. And so that's what motivated us to start Dust on that very simple hypothesis to try to build a product that will unlock the usage of AI at work. The first version of Dust was based on two simple hypotheses. First, the access to the company infrastructure to get access to data and also to be able to take action. So it's a lot of infrastructure work. And the second hypothesis, which was a product hypothesis, was that it's not one assistant, but it's many assistants. One assistant will never be great at every task because it has access to too much data, it's not perfect, the retrieval is not perfect, not ideal, the model is not perfect. And so you get much better result by defining much narrower assistants that will be specialized on the task. And DUST is a product that lets you do that, lets everybody create specific assistants for their use case. And I think those two hypotheses can be framed in a much broader way, which is that today to deploy AI in the world, there's probably an incredible amount of work to be done on the infrastructure part, which is building the pipes to give access to those models to the data and the actions they can take. And there's a humongous amount of work to be done on the product side because we are only scratching the surface. Obviously, the conversation interface is interesting, but it's definitely not the end. So if I want to leave you with one takeaway, I think it will be that there is no better time to be building product and infrastructure in AI. And I think it's actually very simple. You just take your job, look at what you're doing, and there's something that takes you 30 minutes, not more, something like a few tens of minutes. Just try to automatize or to augment it, whether it's using ChatGPT or using DUST or just writing code. Just start on the task you're doing today. And there is an infinite amount of possibilities of what we can do by focusing on that product layer. Thank you. Hey! Hey! Hey!