From AlphaGo to AGI ft ReflectionAI Founder Ioannis Antonoglou

Go is a complex game. There was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforced learning is the answer based on everything that we could measure and everything we could see. But the thing about these systems is that they're not like classic computers where you just know that they always produce the same answer. They're stochastic, they are creative and they have some blind spots. They hallucinate similarly to how Model LLM scholar snakes. So you need to just like really push them and just like see exactly where they break. And the only way you could actually do that is by having like the best humans playing against them.

Today we're excited to welcome Giannis Antonoglo, a researcher and an engineer who has contributed to some of the most significant breakthroughs in AI. As a founding engineer at DeepMind, Giannis played a crucial role in developing AlphaGo, which made history by defeating GO world champion Lee Sedol. He later co led the development of Muzero, which pushed the boundaries even further by mastering multiple games autonomously. Now, as he embarks on his latest venture with Reflection, he's focused on building the next generation of AI agents. We're excited to talk to Giannis about the breakthrough moments in AI history that he's witnessed firsthand, from AlphaGo's famous Move 37 to his perspective today on what's next for the combination of reinforcement learning and large language models. On the way to AGI.

Yannis, thank you so much for joining us today.

Thank you so much for having me.

Yannis, you have an incredible background, Having worked at DeepMind as a founding engineer for over a decade, starting with some of the most notable projects that have really defined the industry. DeepMind quite notably created this notion of building AI within games to start. Can you share a little bit more about why DeepMind chose to start with games at the time?

Yeah. So DeepMind was the first company to truly embrace the concept of artificial general intelligence, or AGI. From the outset, they had grand ambitions, aiming to build systems that would match or exceed human intelligence. So the big question was, and still is, how do you build AGI? And more importantly, how do you measure intelligence in a way that allows for meaningful research and performance improvements? So the idea of using video games as a testing ground came naturally to DeepMind founders. They it was Demis Wassabis and Shane Lake, because Demis had a background in the gaming industry and Shane's PhD thesis defined AGI as a system that could learn to complete any task Feedback games provided a controlled yet complex environment where these ideas could be explored and tested.

And to what extent you mentioned games are, they provide a very controlled environment. To what extent are games representative or not of the real world? Like, if you have a result in games, do you think that generalizes naturally to the real world or not?

So, I mean, I guess games have indeed been viable for developing AI and you actually have like a few examples of that. So you can see that PPO, for example, which is currently being used in RLHF, was developed using OpenAI Gym and Mukoko and Atari. And similarly we have like mcts, which was developed, which stands for Monte Carlo Tree Search, and was developed through board games like Backgammon and Go. But at the same time, games have a number of limitations. So the real world is messy, it's unbounded, and it's a much tougher nut to crack than even the most complex games. So even though it just gives you an interesting testbed to develop new ideas, it's definitely limiting and it doesn't really capture all the complexity of the real world.

Okay, interesting though. So but a lot of the techniques and algorithms that you've developed, the game environment, dpo, et cetera, these are used in the real world.

Yeah, so PPO is actually like exactly what Chess PD used for RLHF and so mcts, it's used in Musero and Musero has been used in the real world in things like compression, video compression for YouTube. It was part of the self driving system at Tesla at some time and it was also used for developing a pilot that was completely controlled by an AI. So yeah, I mean you can see methods like that being used in the real world to solve real problems.

So interesting.

Yannis, I remember back in 2017 when AlphaGo, the movie came out and it featured the incredible game of AlphaGo against Lee Sedol. Can you take us back to that moment in time and maybe the years leading up, up to it, as you're building AlphaGo, how was AlphaGo specifically chosen as the game to focus on?

So I think games have always been a benchmark for AI research. So like before Go, you had chess and chess was like a major milestone with like IBM's Deep Blue defeating Gar Kasparov in the late 90s. And I mean, even though, you know, chess and Go are completely different games and Go is definitely a different beast, there is like games have always been acted as testbeds for the development, especially board games, for the development of new AI methods, actually even going Back to the earliest days of AI research. Turing and Shannon, they both worked on their own versions of chessbots. So now the thing about Go is that it's a much harder problem than chess. And the reason for that is because it's almost close to impossible to define an evaluation method, a heuristic. So in chess, you can just take a look at the board, you can count the number of pawns that each side has, you can see what the ranks of these pawns are, and then you can just make some. You can draw some conclusions on who is winning and why. But, like, in Go, there's nothing like that. It's mostly human intuition. And if you ask a goal, you know, professional player, like, how they know whether. Whether a position is a good one or a bad one, they will say that, like, you know, after having playing the game for so long, they can just, like, feel it in their gut that, like, this is a better position than the other one. So now it's actually a question of how do you encode the feeling in your gut into, like, an AI system? Right. So this is exactly the reason why solving Go was considered the holy grail of AI research for a long time. And it was a challenge that seemed almost impossible, but at the same time, it was within reach. People felt that they could actually get it cracked. And this is exactly what AlphaGo did back in 2016, and it kind of showcased two new methods, which is like deep learning and reinforcement learning. Because back in 2015 and 2016, now we kind of think of deep learning and reinforcement learning as mature technologies, but back then were kind of like, literally like making the. They're taking their first steps, and they were kind of like the new kid in the block. And most people were kind of like, really skeptical about them. Like, everyone thought that deep learning was another AI fad that would just, like, won't last the test of time. So, Yeah, I mean, AlphaGo was chosen because it was, like, clear to show Jace that you actually have, like, the most. The most performant agent in the world. You could actually evaluate it. You can have it play with other humans. And at the same time, it was within reach. Given the latest developments in deep learning and reversal learning.

I remember reading that there's more configurations of the Go board than atoms in the universe by many orders of magnitude. And that blew me away because I grew up playing Go, and it felt like such a. It's a very simple in terms of the rules, but I see why it was the Holy Grail, maybe. Can you explain how AlphaGo worked Technically, maybe explain it to me like I'm a fifth grader, because that is effectively my level of sophistication understanding these things. But how did it work? And you mentioned both reinforcement learning and deep learning were involved. I'd love to peel that back a little bit.

Yeah, absolutely. So AlphaGo has two deep neural networks. So, like, a neural network is a function that takes something as an input and produces something as an output. And it's literally like a black box. We don't really know exactly how it does it. Just know that you can actually, if you train it on enough data, it will just like learn the mapping, it will learn the function from input to the output space. So AlphaGo actually had access to two deep neural networks, the policy network and the value network. And the policy network suggested the most promising move. So it will just take a look at a current port position and just like, say, okay, you know, based on the current position, this is the list of moves adults recommend you just, like, consider playing. And it also had access to the volume metro. We'll just take a look at, like a board position and just like, give you a winning probability, like, what are your chances of actually winning the game? Starting from this position. This is exactly the gut feeling, like, it had, like, its own gut feeling on, like, whether the position is a good one or a bad one. So once you have access to these two networks, then you can actually, like, play in your imagination a number of games. You can consider, like the most promising moves, then you can consider your opponent's most promising moves, and then you can just evaluate its moves, like the value network. And then you can use a method called minmax. What that says is that I want to win the game, but I also know that my opponent wants to win the game. So I want to just pick a move that will maximize my chances of winning, knowing that my opponent will try to maximize their chance of winning. So if you actually do that and simulate a bunch of moves, then you can just get the optimal action. And the way to just do this imagination, this planning, this search in the most efficient way is by using a tree sets method called Monte Carlo tree sets, so MCTs. So whenever people talk about MCTs, they literally just mean this heuristic of how do I, you know, how do I choose which features to consider so that I can make informed decisions. The role for reinforcement learning and deep learning and building AlphaGo was that AlphaGo first of all was a success of reinforcement learning and deep learning, because this is exactly the two methods that powered Alphago and the policy network was initially trained on a large data set of human games. So you had many games played by human professionals and you just consider every position and you consider the move they took at this position. And then you have deep neural network that tries to predict this move. Then once you have the policy network, you need to somehow find a way to just obtain a value network. So we did it in two ways. First, we just took the policy network and we had it play against itself. And we used reinforcement learning to improve the blank strength of the model. So we use a technique called policy gradient. So what policy gradient does is that it just looks at the game and then it looks at the outcome. This is the simplest version, kind of like of policy gradient. It looks at the outcome of the game and for all the moves that led to a win, they'll just like say great, just increase the probability of choosing this move. And for all the moves that led to a loss, it says great, now decrease the probability of like this move being selected in the future. And if you do that for many games and for long enough, then you just get an improved policy. Now once you have this improved policy, you can just generate a new data set of games where the policy plays against itself. And then you have a huge amount of games where for each position you know who the final winner was. So then you can take this network, you can take another network, a value network, and have it predict the outcome of the game based on the current position. So what the network will learn is that if I start at this position and I play under my current policy, on average this is the player who wins. It's either a black player or they're a white player. So this is the first version of a value network. And you can just use it within AlphaGo by combining it with the policy network.

And what were some of the biggest challenges in building this and how did you overcome them?

Yeah, so AlphaGo was not just a realistic challenge, but was mostly, I'd say, an engineering marvel. It was the early versions run on 1200 CPUs and 176 GPUs. And the version that played against listed old used 48 TPUs. So like TPUs were like the first accelerator, custom accelerators. And these accelerators were like really primitive back then because literally it was like the first version, right? Like now the later accelerators are much, much better and much more stable. So the system had to be highly optimized to minimize latency, maximize throughput. We had to build large scale infrastructure for training these networks. And it was a massive endeavor. It just required a lot of coordinated effort from many talented individuals working on different aspects of the project. But, you know, I just like walked you through a number of steps, just like obtain the policy network and the value network. And each of these steps had to just be implemented at the, at the limits of what was available and what was possible back then in terms of scale. And it had to be implemented in a way where people could just tinker with it. They could just try the research ideas fast and get results fast. So, yeah, lots of people scale at levels that hadn't been implemented before. And it's kind of like working at the forefront of for what was possible back then.

I love your highlight of it being a research marvel and an engineering marvel. And I remember you sharing one time that part of the reason this project came about also was because Google had TPUs that they needed a test customer for. And that was the spark this AlphaGo project. So that's pretty incredible.

How much conviction did the DeepMind team have that this is going to work? You mentioned that, you know, at the time, deep learning, reinforcement learning were still relatively novel, but DeepMind was very much founded with that belief. But did you guys think that you were going to be able to have kind of these superhuman level results beating the top Go player in the world? Was it a crazy idea and maybe it'll work, or did the team have conviction like, this is going to work?

Yeah. So at Sehatech, the team had a cautious optimism. So one of AlphaGo's lead developers, Ajah Huang, he is a strong amateur Go player and he had been working on Go for like a decade before AlphaGo happened. And we also had like a lead report of computer game of computer players, and you could see that AlphaGo was significantly stronger than anything that had come before. But Go is a complex game and there was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforced learning is the answer based on everything that we could measure and everything we could see. But that's the thing about these systems is that they're not like classic computers where you just know that they always produce the same answer. They're like stochastic, they are creative and they have some blind spots. They hallucinate similarly to how model LLMs hallucinate. So you need to just like really push them and just like see exactly where they break. And the only way you could actually do that is by having like the best humans playing against them.

Move 37. Can you tell us what that was? It was such a monumental move, and I think everyone watching it at the time was. And Lisa Doll maybe primarily was confused by that move. What was going on in your head when that happened?

So, yeah, I mean, move 37 in game two against Lizzie Doll was literally a spectacular moment in the sense that it kind of showed gaze to the world, that AlphaGo has creativity, and it demonstrated that AI could come up with strategies that even top human players hadn't considered. So at first, like, I still remember that, like, we thought that alphago made an error so that it actually hallucinated it did something that it didn't mean to do. But then it turned out to be a brilliant, conventional move that underscored that the system had a deep understanding of the game, that the system actually had creativity. It could think of things that people hadn't thought of before.

I want to take us to another key move in the game. I think it was in Game four. At this point, I was rooting for Lee because I was like, the poor guy needs to win a game.

Move 78.

I think alphago made a mistake and Lee Sedol notices it, I guess. What was the weakness there that Lee found during the game?

Yeah, exactly. So, I mean, Lisa Doll's fixture in game four was literally a testament to human ingenuity. Like move 78 was unexpected and caught AlphaGo off guard. Initially, AlphaGo, based on its evaluations, misinterpreted as a mistake and thought that it was actually like, winning. So that's why it didn't respond appropriately. And this kind of highlighted the blind spot in the system. So the game showed that While systems like AlphaGo are extremely powerful, at the same time, they still have vulnerabilities and there were still areas where we could further improve it.

But how do you go about improving something like that? Do you need to show it a lot more data of kind of that type of human ingenuity move, or how do you go about fixing and patching those?

So, yeah, I mean, it's actually interesting that by the end of the games, like Lucidol, we just put together a benchmark where you're just kind of like trying to quantify and just have a way of measuring the mistakes that like, AlphaGo makes and these kind of blind spots, let's say. And then I just write a number of approaches to just, like, improve the algorithm so that we can solve these issues. And what happened is that actually the most effective way of getting rid of them was just like, do what we were doing just like at higher scale and better, so just like change the architecture of the model. We just switched to a deep breast net with two output heads and we also just had a bigger network trained on more data, then just moved to AlphaZero and spread the algorithms and that of like made it so that we didn't have any hallucinations anymore. So in a way, just like scale data, you know, things that are always kind of the well known recipe in the field of AI is exactly what solves it in our case too, with scale and data.

How much did higher quality data, or maybe specifically data from great professional players, the best professional players, make a meaningful difference or was it just any data?

Now for us, what mattered was that we kind of solved it using self play. So we actually had access to the most competent go player in the world and we just used it to generate the best quality games and then we just trained on these games. So I guess we didn't need to have human experts because you had like an expert in house. It wasn't human.

Huh.

Interesting. Amazing. Well, I'd love to move on to the progression from AlphaGo to AlphaZero. And you talked a little bit about this notion of self play just now. AlphaZero was powerful because it learned how to play the game from scratch entirely from self play without any human intervention. Can you share more about how that worked and why that was important?

So AlphaZero was a game changer because it learned entirely from scratch with through self play without any human data. And this was like a major leap from AlphaGo because AlphaGo, as I said, relied heavily on human expert games. So two things happened. First of all, AlphaZero managed to simplify the training process and also showed that AI could literally just get from zero to superhuman performance just purely by playing against itself. And that allowed it to just be applicable to a whole range of new domains that were out of reach because there weren't enough human data for it. But I think the more important thing is that we just saw that AlphaZero also solved all the issues that AlphaGo had in terms of hallucinations, in terms of blind spots and robustness. So alphazeros was a better method, just full star.

And you explained kind of how AlphaGo worked to a fifth grader. What would you tell the fifth grader would be the key difference technically that you implemented with AlphaZero.

So AlphaZero, just like AlphaGo, uses a policy network and a value network along with multi calculuses. So in that respect it's exactly the same as AlphaGo. So the key difference is in training. AlphaZero starts with random weights and lands by playing games against itself. And by playing games against itself, it iteratively improves its performance. But the main idea behind AlphaZero is that whenever you take a set of weights, a set of policy and value networks and then you just combine them with search, then you just end up with a better player. You just increase your performance, you just become a stronger player. What that meant is that we can actually use this mechanism to improve the, the model policy, the role policy. So this is what we call in reinforcement learning, a policy improvement operator. Whenever you can just take an existing policy and then do something, some magic, and then just come up with a better policy. And then you can just take this policy and distill it back to the initial policy and then just repeat this process, then you have a reinforcement learning algorithm. And I think this is exactly what people are trying to do today with you know, Q or like, you know, synthetic data. This is exactly the idea of like, how can I take a policy, do something with it, planning, search, compute whatever it is and derive a better policy which I can then imitate and just like kind of distill back to the original policy. So this is exactly what AlphaZero is doing. It uses MCTS search to produce a better policy. Then it takes its trajectories, it trains this policy and value network on the new better trajectories and it repeats the this process until it converges to an expert level Go player.

That's fascinating and counterintuitive. That kind of like starting without the weights that you would have from professional level players is actually a better starting place.

The epitome of AI agents in games was achieved I think via Muzero, which is the progression even from AlphaZero itself. And it's also where you became one of the co leads or one of the leads of, of the game. AlphaZero was obviously impressive because of self play, but it also needed to be told the environment's dynamics or the rules of the game. And MU0 takes this to the next level without needing to be told the rules of the game. And it mastered quite a few different games. Go chess and many others. Can you share a little bit about how mu0 worked and why was this particular particularly meaningful?

Absolutely. So AlphaZero, as you said, was a massive success in games like chess. Go Shoigi. So in games where we actually had access to the game rules, where we actually had access to a perfect simulator of the world, but like this reliance on the perfect simulator made it challenging to apply it to real world problems. And real world problems are often messy and they lack clear rules. And it's really hard just like write a perfect simulator of them. So that's exactly what MU0 tried to solve. So MU0 masters the games, of course, like Go Chess and Sogi, but it also masters more visually challenging games or games that like a hot goat like Atari. And it does that without having access to the simulator. It just learns how to build an internal simulator of the world and then just use this internal simulator in a way similar to what AlphaZero is doing. So it does that by using model based reinforced learning, where what that means is that you can just take a number of trajectories generated by an agent and then try and learn a model, learn a prediction model of how the world works. So this is actually quite similar to what methods like SORA are trying to do now, where they just take YouTube videos and they try to just learn a world model by just trying to predict based on starting from one frame, what's going to happen in the future frames. So new zero tries to do exactly that. But it does it in a way different from generative models in the sense that it tries to only model things that matter for solving the reinforcement learning problem. So it tries to predict what the rewards are going to be in the future. What's the value of future states? What's the value of what's the policy for future states? So only things that you need to think your mcts, but the fundamentals kind of remain the same. So how do you just learn a model based on trajectories and then once you have this model, you can just combine it with search and get superhuman performance. So of course you can always decouple the two problems and have the model being trained separately from data out in the wild, and then just combine that with mu0. And we just found that back then, given the limitations of our models and the smaller sizes, kind of made more sense to just keep those two together and only have the model predict things that matter for planning instead of just try to model everything because you're kind of hitting the limits of what the capacity of the model could take.

So interesting.

Is it right to assume then that not only SORA takes the same approach, but maybe other world models or other robotics foundation models?

Yeah, so anything that tries to just like build a model of how the world works and then just like use that for planning, it's within new zero like methods. So yeah, you can just like train it on YouTube videos. You can train it on like the inputs coming from like robots. You can train it on you know, any, any environment. You can even think of like large language models as a form of models of like text. So the model text, but the thing about text is that the model is a bit trivial. You don't need to just. There aren't many artifacts happening when you're trying to predict what the next world is going to be.

Have you seen the ideas behind Muzero kind of be used outside gameplay or in messy real world environments?

Yeah, I mean so as I've said AlphaZero and use your quite general methods. And there's a number of scientific communities in chemistry, so there's alpha chem in quantum computing. Some people try to use Alphazero in optimization where they just like adopted AlphaZero because it was really powerful in really doing, planning and just like solving these optimization problems at the same time. U0 was incorporated in a version of like Tesla's self driving system. It was kind of like reported in their AI day and it was also used and I think it's currently being used within YouTube as a custom compression algorithm. But I think you know, it's early days and takes time for like these new technologies to be fully adopted from by the industry.

We'd love to talk a little bit more about reinforcement learning and agents. You alluded earlier to the fact that reinforcement learning and deep learning back in 2015 were new nascent ideas. They really grew in popularity, 2017, 2018, 2019 onwards. And then they were overshadowed by LLMs largely because of the GPT and everything else that came out. But now reinforcement learning is back. Why do you think that is the case?

Yeah, I mean first of all, LLMs and multimodal models have indeed brought incredible progress to AI. So these models are exceptionally powerful and can perform some truly impressive tasks. But they have some fundamental limitations and one of them is the availability of human data. People just keep talking about the data wall and what happens once you run out of high quality data. And this is exactly where reinforcement learning shines. So reinforcement learning excels because it doesn't rely solely on pre existing human data. Instead reinforcement learning uses experience generated by the agent itself to prove its performance. So this self generated experience allows reinforcement learning to learn and adapt and to even adapt to scenarios where human data is scarce or non existent. So if you define the reinforcement learning problem in the right setting, in the right way, you can literally effectively exchange compute for intelligence. You can just get to A point similar to where we were with AlphaZero, where we just, the moment we threw more computer at it, we made the networks bigger, we just used more games, we just literally got a better player. And it was deterministic. You always get a better player. So I guess this is exactly where we want to be with these synthetic data pipelines. Currently we have that with the scaling clause in LLMs that if you have more data and bigger models, then you can predict that there's going to be an improvement in performance. But once you've run out of human data, how do you just keep going? And synthetic data is the answer to that. And the only way that you can actually get high quality reinforcement learning, high quality data to just improve your model is via some form of reinforcement learning. And I'm just keeping reinforcement learning as a really kind blanket term here, where I just define it as anything that lends through trial and error.

How do you think reinforcement learning is being brought into the LLM world? And you mentioned Q earlier. I guess in a closed form game you have a pretty clearly defined policy and value function. How does that work in a messy real world environment or the LLM world?

I mean, I guess there are two different types of messy real world, right? Like there is if you try to just like build a controller or something, that's a really messy environment. And then if you operate in the digital space. So personally I believe that digital AGI will just happen much earlier than robotics AGI. And the reason for that is exactly that you have control over the environment. And the environment computer is like the digital world. So even though it's messy and it's noisy, it's still contained. It's not like the real kind of world in that sense. So now in terms of how do you bring reinforcement learning? So reinforcement learning is, we used to say in DeepMind that you have the problem and you have the solution. And the problem setting of reinforcement learning is how do I take a model, how do I take a policy and generate synthetic data? Or like I find a way to improve this policy by interacting with the environment via trial and error. And this is like the reinforcement learning problem setting, right? And then there's like the solution space where you have value functions and have like reinforcement learning methods. So I think that there's a lot of inspiration to draw from, like classical reinforcement learning methods that were developed in the past decades, but you have to adjust them to the new world of LLMs. So methods like QStar try to do that by just taking the idea that if I Have a policy and then I do planning. I consider possible future scenarios and then I have a way to evaluate which one is better. Then I can just take the best ones and then ask the model to imitate these better ones. And this is like a way of improving the policy. So in the classic RL framework you do that by using a policy and a value network. In the new world you'll just do that by having a reward model or asking your LLM to just give you feedback on an output it gave you.

So interesting.

You also talked a little bit about synthetic data earlier. I think some folks are very bullish on synthetic data and some folks folks more skeptical. I also believe that synthetic data is more useful in some domains where outcomes and success is perhaps more deterministic. Can you share a little bit about your perspective on the role of synthetic data and how bullish you are on it?

Yeah, I mean I think synthetic data is something that we have to solve one way or another. So it's not about whether you're bullish or not. It's an obstacle. But have just find a way around it. Like we will run out of data. Like you know, we. There is so much data like humans can produce. And also like it's important that these systems start taking actions. They start learning from their own mistakes. So we need to just find a way to make like synthetic data work. Now what people have done is that they've tried like the most I guess like naive approach where you just like take the models, they produce something and you try to just train on that. And of course they've seen that there's mode collapsing and this just doesn't work out of the box. But new methods never work out of the box. You just need to invest in it and just take your time and really think of what's the best way of doing it. So I'm really optimistic that we'll just definitely find ways to improve these models. And I think that actually there is a number of methods out there like the two star and equivalence that just in the new world where people don't really share their research breakthroughs the way they used to is probably hidden behind some company trade secrets.

I want to ask about reasoning and novel scientific discoveries. Do you think that that can kind of naturally come out of just scaling LLMs if you have enough data or do you think that kind of like the ability to reason and come up with net new ideas requires kind of doing reinforcement learning and deeper compute at inference time?

So I think you need reinforcement learning to get better Reasoning because the distribution of it's also about the distribution of data. Right. You have a lot of data out in the wild in the Internet, but at the same time you don't always have the right type of data. So you don't have the data where someone reasons and they just explain the reasoning in detail. You have some of it and it's incredible that the models you've actually managed to pick it up and just imitate it. But if you want to just improve on that capability, then you need to do that through reinforcement learning. You need to just take show the model how this kind of emerging capability can further be improved by just have it generate synthetic data, interact with the environment, just tell it when it's doing something right and when it's not doing something right. So yeah, I think that reinforced learning is definitely part of the answer for that.

AlphaGo, AlphaZero and MU0 are the most powerful agents we've ever built. Can you share a little bit about how some of the lessons and learnings unlocked from that are relevant to how we're pursuing building AI agents today?

Yeah, so I feel like AlphaGo and Me Zero they've actually fundamentally transformed our approach to AI agents because they highlight the importance of planning and scale. In my opinion, that if you actually look at the charts of different models and how they scale, you can see that AlphaGo and AlphaZero were kind of really ahead of their time. Like they were kind of outliers. You had like these curves of like how compute scaled and then you have like AlphaZero or like somewhere standing on its own. So it showed that like if you can scale and you can really push on that, then you can get like incredible, incredible results. At the same time, you know, it also showed that you don't have just only train. You can also like, you know, have better performance during inference, during tests, during evaluation, by just like using planning. And I think that this is something that we will start seeing more and more in the near future or these methods will just start thinking more, planning more before they're just making any decisions. So I'd say that this is more of the heritage of AlphaGo and AlphaZero and E0. It's the basic principles and the basic principles are of that scale matters, planning matters. These methods can really solve problems that we thought that are insanely complex or beyond what we can solve on our own. Similar problems with the ones that you actually observed today with these large language models are things that we saw back then back in 2016, we actually saw that These models can hallucinate or that at the same time they're also creative, that they will just come up with solutions that we hadn't thought of. But they can also have blind spots or hallucinate or be susceptible to kind of like adversarial attacks, which I guess everyone knows now that these neural networks suffer from. So I think that these are the main kind of lessons drawn from this line of work.

What do you think are the biggest open questions from this line of work for the field dancer going forward?

So the main question is we had like AlphaGo and V0 and we just like managed to have like this insanely robust and reliable systems that will just always play Go and at the, you know, the highest possible kind of level and they'll just like achieve consistently. They will just like be top of the leaderboard. We'll just like never lose a game. So Alpha Go Master actually like played against 60 people in online matches and just like literally one in every single one of them. So there was like no. These methods were like incredibly robust, reliable. And I felt like this is exactly what we're missing now with these LLM based agents. Sometimes they get it, sometimes they don't. You cannot trust them. They will just like, you know, you have like some amazing demos but like, you know, they happen once every two times even, or like once every 10 times you have like something amazing. And the remaining nine, they just lost their way and didn't do anything. So I think what we need to do is just find a way to just make these LLM based agents equally robust to the ones that we had with AlphaGo and V0 and AlphaZero. This is like the new open question of how do we actually do that.

We'd love to move into some of your thoughts on the broader ecosystem today. You've touched on a few really core problems that people are working on right now. One, the data wall problem that will hit eventually, perhaps by 2028 or so, as some folks predict. Another being the idea of planning as an area that AI agents need to get better at. And then, you know, a third idea that you just described was around robustness and reliability. Can you share a little bit about maybe some of these areas that you think the whole field needs to solve that you are most excited about to help us unlock this vision of really getting to the AI agents that we want?

Yeah, I mean, I'll just also add another one to the list. So I think another major challenge is how do we improve the in context learning capabilities of these models or how do we make sure that these systems can learn on the fly and how they can adapt to new contexts, like, quickly? So this is like another thing that I think is going to be really important. It's going to happen the next few years, couple of years, actually.

So, Yanis, what's the term that you used for that?

In context learning?

In context learning.

In context learning, yeah. So it's the idea that a system can actually learn how to do a new task with like, few shot prompting. Like, it kind of like sees a few examples and on the fly, it kind of like learns how to adapt to the new environment, it learns how to use the new tools that were provided to it. Or like it's kind of like learns it's not just all the knowledge it is stored in swage, but like, it can also acquire new knowledge by just like interacting with the real world, interacting with the environment. So I think that this is like another place where there is a lot of work happening at the moment and we're going to have amazing progress in the next couple of years. And I'm really excited about that. So, yeah, I mean, to recap, I think planning is important in context learning is important and reliability. So the best way to achieve reliability is just ensure that this model somehow know how to return from their mistakes. So if they just made a mistake somewhere, they can just see that and they're like, okay, I made a mistake, I'll just correct for it the way that humans make mistakes all the time, but you can correct for them. So these are the three areas which I'm really excited to see progress on.

Now that you've kind of embarked on your own entrepreneurial journey, how do you think that the areas where startups can compete against the big research labs? And how do you kind of motivate yourself for that journey?

Yeah, I mean, it's completely like, it's a new world for me, but at the same time, it's not that new because when I joined DeepMind, it was literally a startup and I was like, literally in the first few employees. So I actually saw that firsthand. But one of the benefits of working for a startup is that the agility and the focus, so everyone really cares. Everyone just moves really fast and there's like a clear focus on what we want to build. So the building is like, what's the most important kind of motivation for people? Just like building. And I think that this is one of the big advantages that startups have over more established businesses. At the same time, it's easier to Just pivot to adapt to new findings, new technologies. You're not kind of like tied to some pre existing solutions or some products that you don't want to deprecate because they bring a lot of revenue to you. While if you're a startup you have no sad chains. You can just move fast and be innovative and just break conventions and at the same time just allows you to leverage open source resources, things that are out of touch for the big labs and you don't have the red tape that big places tend to have.

I love the term that you use sometimes, Jonas. Main quest versus side quest.

Yeah, it's the idea of having a main focus in big places. In big labs they have many different projects that people are working on and it usually happens that they have the main quest, the main thing that everyone's working on. And there's many multiple smaller side quests that the idea is to just feed into the bigger quest. But usually they don't get as many resources or as much focus from the leadership, so they tend to atrophy in the broader field.

What are some of the most defining projects that you admire the most and maybe who are some of the most influential researchers that you admire the most?

Yeah, absolutely. So, so I actually like started my AI research journey back in 2012 and I've actually like seen some milestones. So I'll just like, I give a list of like what I think are like the main milestones like in AI in the past like 12 years that I've been around. So the first one I'll say is like Alexnet, this is the first paper that kind of showed that deep learning is the answer. I mean back then it didn't feel like it, it just like felt like, you know, a kind of curiosity. But like now I think that most people are convinced that like deep learning is part of the answer. Then it was tqn. I had the pleasure to actually work on TQN and just like see it firsthand how it started. It was actually developed by a friend of mine, Vlad Me, flagged me. And it was like the first system that showed that you can actually combine deep learning with reinforced learning to achieve human performance or like superhuman performance in really complex environments. Then this was AlphaGo again. I was really lucky to just work on that. And it showed that scale and planning are really important ingredients and if you just do that right then you get huge success in an incredibly complex environment. AlphaFold, another one. This is again by DeepMind. It shows that these methods are not just things that you can use to solve games, but they actually will make this world a better place. They will just ensure that healthcare is improved, that scientific discoveries are being realized, that we'll just like make sure this world is a better place by using AI then ChatGPT, it kind of like brought AI to everyone, just like made it accessible to the broad audience like everyone knows what AI is now. It has made my life of explaining my job much easier. So and finally GPT4 and I think that probably GPT4 is like the latest kind of big advancement in AI because it kind of showed that artificial general intelligence is a matter of years, it's within reach. Yeah, we are getting there. I think that most people now believe that we are a few years away from AGI and that's because of the incredible breakthrough that GPT4 was. Now in terms of some people I really admire before I forget, so I'd say first David Silver, he was my PhD supervisor, he was my mentor at DeepMind. He's an incredibly AI researcher. He led Alphco and AlphaZero and, and you know, he is, you know, he has a gilding dedication to the field of reinforcement learning and he's you know, probably the one of the smartest people or maybe the smartest person I know and amazing guy in amazing reinforcement learning engineer. And the second one I'd say is Ilya Sutskever. And you know, he was a co founder of OpenAI. I had the opportunity to work with him just a little bit in the really Early Days of AlphaGo. But I think his commitment to scaling a methods and pushing the boundaries of what these systems can achieve is remarkable and he made sure that GPT3 and GPT4 happen. So yeah, immense respect towards him.

Thank you for sharing that.

Let's close out with some rapid fire questions. Maybe first, what do you think will be the next big milestones in AI, let's say in the next one, five and 10 years.

So I think the next five to 10 years the world would be a different place. I actually really believe that. I think that the next few years we'll see models becoming powerful and reliable agents that can actually independently execute tasks. And I think that AI agents will be massively adopted across industries, especially in science and healthcare. So in that sense I'm really excited on what's coming in AI and what I'm most excited about is AI agents systems can actually do tasks for you. And this is exactly what we're building at reflection.

In what year do you think we'll pass the 50% threshold on SUI bench.

So I think we're one to three years away from the 50% threshold for SWEE agents and three to five years from achieving 90%. So the reason is, while progress is amazing, I think we still need reliable agent to hit these milestones. And it's really. When it comes to research, it's hard to make precise predictions.

When do you think we'll hit the data wall for scaling LLMs? And do you think all the research in RL is mature enough to keep up our slope of progress, or do you think there'll be a bit of a lull as we try to figure out what happens when we hit the wall?

So I think, like, the wall, you know, based on like, what I've read, I think like, we have at least one more year for text, just like before we hit the wall, and then we have like these extra modalities which might actually buy us maybe a year extra. And I think we are in a really good place to just like, start using synthetic data. So in the next few years we'll just figure out the synthetic data problem. So I think that we won't really hit the wall. Just like we'll hit the wall, but no one realizes it because we'll have new methods in place.

Do you think LLMs will have their AlphaGo moment and if so, when?

I think LLMs had their AlphaGo moment with the initial release of ChatGPT, where they showcase the power and the progress made over the past decade. I feel like what they hadn't had yet is their AlphaZero moment. And that's the moment where more compute directly translates to increased intelligence without human division. And I think this breakthrough is still on the horizon.

When do you think that'll happen?

I think it's going to happen in the next five years. Wow.

Amazing. Yanis, thank you so much for joining us and taking us through the awesome History of AlphaGo, AlphaZero MU0, your own journey through DeepMind, and then many of the core research problems that the whole industry is tackling today around data and building for reliability and robustness and planning and in context learning. We're really excited for the future that you're helping us build and that you're pushing forward in the field as well. So thank you so much.

Jonas, thank you so much for having me.