David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning | Lex Fridman Podcast #86

The following is a conversation with David Silver, who leads the Reinforcement Learning research group at DeepMind and was the lead researcher on AlphaGo, AlphaZero & Co led the Alpha Star and MU0 efforts and a lot of important work in reinforcement learning in General. I believe AlphaZero is one of the most important accomplishments in the history of artificial intelligence and David is one of the key humans who brought AlphaZero to life together with a lot of other great researchers at DeepMind. He's humble, kind and brilliant. We were both jet lagged but didn't care and made it happen. It was a pleasure and truly an honor to talk with David. This conversation was recorded before the outbreak of the pandemic. For everyone feeling the medical, psychological and financial burden of this crisis, I'm sending love your.

Way. Stay.

Strong. We're in this together. We'll beat this thing. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcast support on Patreon, or simply connect with me on Twitter lexfriedman spelled F R I D M A N. As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the.

Conversation. I hope that works for you.

And doesn't hurt the listening experience. Quick summary of the Ads2 sponsors masterclass and Cash App Please consider supporting the podcast by signing up to masterclass@masterclass.com lex and downloading cash App and using Code Lex Podcast. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use Code Lex Podcast. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating. I recommend Ascent of Money as a great book on this history. Debits and credits on ledgers started around 30,000 years ago, the US dollar created over 200 years ago, and Bitcoin, the first decentralized cryptocurrency released just over 10 years ago. So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to and just might redefine the nature of money. So again, if you get Cash App from the App Store, Google Play and use the code lexpodcast, you get $10 and cash app will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education.

For young people around the.

World. This show is sponsored by Masterclass. Sign up on masterclass.com lex to get a discount and to support this.

Podcast. In fact, for a limited time.

Now, if you sign up for an all access pass for a year, you get to get another all access pass to share with a.

Friend. Buy one, get one.

Free. When I first heard about Masterclass, I thought it was too good to be true. For $180 a year, you you get an all access pass to watch courses from to list some of my favorites. Chris Hadfield on space exploration. Neil DeGrasse Tyson on scientific thinking communication. Will Wright the creator of SimCity and Sims on game design. Jane Goodall on conservation. Carlos Santana on guitar. His song Europa could be the most beautiful guitar song ever written. Garry Kasparov on chess. Daniel Negreano on poker and and many, many more. Chris Hatfield explaining how rockets work and the experience of being launched into space alone is worth the money for me. The key is to not be overwhelmed by the abundance of choice. Pick three courses you want to complete. Watch each of them all the way through. It's not that long, but it's an experience that will stick with you for a long.

Time. I promise. It's easily worth the.

Money. You can watch it on basically any device. Once again, sign up on masterclass.com lex to get a discount and to support this podcast. And now, here's my conversation with David.

Silver. What was the first program you ever written and what programming language do you.

Remember? I remember very clearly. Yeah, my parents brought home this BBC Model B microcomputer. It was just this fascinating thing to me. I was about 7 years old and couldn't resist just playing around with it. So I think first program ever was writing my name out in different colors and getting it to loop and repeat that. And there was something magical about that, which just led to more and.

More. How did you think about computers back then? Like the magical aspect of it, that you can write a program and there's this thing that you just gave birth to that's able to create sort of visual elements and live on its own? Or did you not think of it in those romantic notions? Was it more like, oh, that's cool, I can, I can solve some.

Puzzles? It was always more than solving puzzles. It was something where, you know, there was this limitless possibilities. Once you have a computer in front of you, you can do anything with it. I used to play with Lego with the same feeling. You can make anything you want out of Lego, but even More so with a computer, you're not constrained by the amount of kit you've got. And so I was fascinated by it and started pulling out the user guide and the advanced user guide and then learning. So I started in BASIC and then later 6502. My father also became interested in this machine and gave up his career to go back to school and study for a master's degree in artificial intelligence, funnily enough, at Essex University when I was seven. So I was exposed to those things at an early age. He showed me how to program in Prolog and do things like querying your family tree. And those are some of my earliest memories of trying to figure things out on a.

Computer. Those are the early steps in computer science.

Programming. But when did you first fall.

In love with artificial intelligence or were the ideas the dreams of.

AI? I think it was really when I went to study at university. So I was an undergrad at Cambridge and studying computer science. And I really started to question, what really are the goals? What's the goal? Where do we want to go with computer science? And it seemed to me that the only step of major significance to take was to try and recreate something akin to human intelligence. If we could do that, that would be a major leap forward. And that idea certainly wasn't the first to have it, but it nestled within me somewhere and became like a bug. I really wanted to crack that.

Problem. So you thought it was like.

You had a notion that this is something that human beings can do, that it is possible to create an intelligent.

Machine? Well, I mean, unless you believe in something metaphysical, then what are our brains doing? Well, at some level, they're information processing systems which are able to take whatever information is in there, transform it through some form of program and produce some kind of output which enables that human being to do all the amazing things that they can do in this incredible.

World. So then, do you remember the first time you've written a program that, because you also had an interesting games. Do you remember the first time you were in a program that beat you in a game or beat you at anything, sort of achieved super David Silver level.

Performance? So I used to work in the games industry, so for five years I programmed games for my first job. So it was an amazing opportunity to get involved in a startup company. And so I was involved in building AI at that time. And so for sure there was a sense of building handcrafted, what people used to call AI in the games industry, which I think is not really what we might think of as AI in its fullest sense, but something which is able to, to take actions in a way which makes things interesting and challenging for the human player. And at that time I was able to build these handcrafted agents which in certain limited cases could do things which were able to do better than me, but mostly in these kind of twitch like scenarios where they were able to do things faster or because they had some pattern which was able to exploit repeatedly. I think if we're talking about real AI, the first experience for me came after that when I realized that this path I was on wasn't taking me towards it wasn't dealing with that bug which I still had inside me to really understand intelligence and try and solve it. Everything people were doing in games was short term fixes rather than long term vision. And so I went back to study for my PhD, which was funnily enough trying to apply reinforcement learning to the game of Go. And I built my first Go program using reinforcement learning. A system which would by trial and error play against itself and was able to learn which patterns were actually helpful to predict whether it was going to win or lose the game, and then choose the moves that led to the combination of patterns that would mean that you're more likely to win and that system beat.

Me. And how did that make you.

Feel? Made me feel.

Good. I mean was there sort of the. Yeah, it's a mix of a sort of excitement and was there a tinge of sort of like almost like a fearful awe? You know, it's like in space 2001 space odyssey. Kind of realizing that you've created something that is, you know, that's achieved human level intelligence in this one particular little task. And in that case I suppose neural networks weren't.

Involved. There were no neural networks in those days. This was pre deep learning revolution. But it was a principled self learning system based on a lot of the principles which people are still using in deep reinforcement learning. How did I feel? I think I found it immensely satisfying that a system which was able to learn from first principles for itself was able to reach the point that it was understanding this domain better than I could and able to outwit me. I don't think it was a sense of awe, it was a sense that satisfaction that something I felt should work had.

Worked. So to me, AlphaGo and I don't know how else to put it, but to me, AlphaGo and AlphaGo Zero. Mastering the game of Go is again to me the most profound and inspiring moment in the history of artificial intelligence. So you're one of the key people behind this achievement, and I'm Russian, so I really felt the. The first sort of seminal achievement when Deep Blue beat Garry Kasparov in 1987. So as far as I know, the AI community at that point largely saw the game of Go as unbeatable in AI using the sort of the state of the art to brute force methods, search methods. Even if you consider, at least the way I saw it, even if you consider arbitrary exponential scaling of computer, Go would still not be solvable. Hence why it was thought to be impossible. So given that the game of Go was impossible to master, when was the dream for you? You just mentioned your PhD thesis of building the system that plays Go. What was the dream for you? That you could actually build a computer program that achieves world class, not necessarily beat the world champion, but achieves that kind of level of play and.

Go. First of all, thank you, that's very kind words. And funnily enough, I just came from a panel where I was actually in a conversation with Garry Kasparov and Murray Campbell, who was the author of Deep Blue. And it was their first meeting together since the match. So that just occurred yesterday. So I'm literally fresh from that experience. So these are amazing moments when they happen. But where did it all start? Well, for me, it started when I became fascinated in the game of Go. So Go for me, I've grown up playing games. I've always had a fascination in board games. I played chess as a kid. I played Scrabble as a kid. When I was at university, I discovered the game of Go. And to me, it just blew all of those other games out of the water. It was just so deep and profound in its complexity, with endless levels to it. What I discovered was that I could devote endless hours to this game. And I knew in my heart of hearts that no matter how many hours I would devote to it, I would never become a grandmaster or there was another path. And the other path was to try and understand how you could get some other intelligence to play this game better than I would be able to. And so even in those days, I had this idea that what if it was possible to build a program that could crack this? And as I started to explore the domain, I discovered that this was really the domain where people felt deeply that if progress could be made in Go, it would really mean a giant leap forward for AI. It was the challenge where all other approaches had failed. This is coming out of the era you mentioned, which was in some sense the golden era for the classical methods of AI, like heuristic search in the 90s, they all fell one after another. Not just chess with Deep Blue, but checkers, backgammon, Othello. There were numerous cases where systems built on top of heuristic search methods with these high performance systems had been able to defeat the human world champion in each of those domains. And yet in that same time period, there was a million dollar prize available for the game of Go for the first system to be a human professional player. And at the end of that time period in year 2000, when the prize expired, the strongest Go program in the world was defeated by a nine year old child when that nine year old child was giving nine free moves to the computer at the start of the game to try and even things up. And Computer GO expert beat that same strongest program with 29 handicap stones, 29 free moves. So that's what the state of affairs was when I became interested in this problem in around 2003, when I started working on Computer Go, there, there was nothing, there was very, very little in the way of progress towards meaningful performance at anything approaching human level. And so people, it wasn't through lack of effort. People had tried many, many things. And so there was a strong sense that something different would be required for Go than had been needed for all of these other domains where AI had been successful. And maybe the single clearest example is that Go, unlike those other domains, had this kind of intuitive property that a Go player would look at a position and say, hey, here's this mess of black and white stones. But from this mess, oh, I can predict that this part of the board has become my territory, this part of the board's become your territory. And I've got this overall sense that I'm going to win, and that this is about the right move to play. And that intuitive sense of judgment, of being able to evaluate what's going on in a position. It was pivotal to humans being able to play this game, and something that people had no idea how to put into computers. So this question of how to evaluate a position, how to come up with these intuitive judgments, was the key reason why Go was so hard, in addition to its enormous search space, and the reason why methods which had succeeded so well elsewhere, failed in Go. And so people really felt deep down that in order to crack Go, we would need to get something akin to human intuition. And if we got something akin to human intuition, we'd be able to solve many, many more problems in AI. So for me, that was the moment where it's like, okay, this is not just about playing the game of Go. This is about something profound. And it was back to that bug which had been itching me all those years. This is the opportunity to do something meaningful and transformative. And I guess a dream was.

Born. That's a really interesting way to put it. So almost this realization that you need to find, formulate Go as a kind of a prediction problem versus a search problem was the intuition. I mean, maybe that's the wrong crude term, but to give it us the ability to kind of intuit things about positional structure of the board now. Okay, but what about the learning part of it? Did you, did you have a sense that you have to. That learning has to be part of the system? Again, something that hasn't really, as far as I think, except with TD gammon and in the 90s with RL a little bit, hasn't been part of those state of the art game playing.

Systems. So I strongly felt that learning would be necessary. And that's why my PhD topic back then was trying to apply reinforcement learning to the game of Go and not just learning of any type, but I felt that the only way to really have a system to progress beyond human levels of performance wouldn't just be to mimic how humans do it, but to understand for themselves. And how else can a machine hope to understand what's going on except through learning? If you're not learning, what else are you doing? Well, you're putting all the knowledge into the system and that just feels like something which decades of AI have told us is maybe not a dead end, but certainly has a ceiling to the capabilities. It's known as the knowledge acquisition bottleneck. That the more you try to put into something, the more brittle the system becomes. And so you just have to have learning. You have to have learning. That's the only way you're going to be able to get a system which has sufficient knowledge in it. Millions and millions of pieces of knowledge, billions, trillions of a form that it can actually apply for itself and understand how those billions and trillions of pieces of knowledge can be leveraged in a way which will actually lead it towards its goal without conflict or other.

Issues. Yeah, I mean, if I put myself back in that time, I just wouldn't think like that without a good demonstration of rl. I would think more in the symbolic AI, like not learning, but sort of a simulation of knowledge base, like a growing knowledge base, but it would still be sort of pattern based, basically have little rules that you kind of assemble together into a large knowledge.

Base. Well, in a sense, that was the state of the art back then. So if you look at the Go programs which had been competing for this prize I mentioned, they were an assembly of different specialized systems, some of which used huge amounts of human knowledge to describe how you should play the opening, how you should. All the different patterns that were required to play well in the game of Go, endgame theory, combinatorial game theory, and combined with more principled search based methods which were trying to solve for particular subparts of the game, like life and death, connecting groups together, all these amazing sub problems that just emerge in the game of Go. There were different pieces all put together into this collage which together would try and play against a human. And although not all of the pieces were handcrafted, the overall effect was nevertheless still brittle and it was hard to make all these pieces work well together. And so really what I was pressing for, and the main innovation of the approach I took, was to go back to first principles and say, well, let's back off that and try and find a principled approach where the system can learn for itself just from the outcome. Like learn for itself. If you try something, did that help or did it not help? And only through that procedure can you arrive at knowledge which is verified. The system has to verify it for itself, not relying on any other third party to say this is right or this is wrong. And so that principle was already very important in those days, but unfortunately we were missing some important pieces back.

Then. So before we dive into maybe discussing the beauty of reinforcement learning, let's take a step back. We kind of skipped it a bit, but the rules of the game of Go, the elements of it perhaps contrasting to chess, that sort of you really enjoyed as a human being, and also that make it really difficult as a AI machine learning.

Problem. So the game of Go has remarkably simple rules. In fact so simple that people have speculated that if we were to meet alien life at some point, that we wouldn't be able to communicate with them, but we would be able to play a game of Go with them, because they probably have discovered the same rule set. So the game is played on a 19 by 19 grid, and you play on the intersections of the grid and the players take turns. And the aim of the game is very simple. It's to surround as much territory as you can, as many of these intersections with your stones and surround more than your opponent does. And the only nuance to the game is that if you fully surround your opponent's piece, then you get to capture it and remove it from the board and it counts as your own territory. Now, from those very simple rules, immense complexity arises. There's kind of profound strategies in how to surround territory, how to kind of trade off between making solid territory yourself now compared to building up influence that will help you acquire territory later in the game, how to connect groups together, how to keep your own groups alive. Which patterns of stones are most useful compared to others? There's just immense knowledge. And human Go players have played this game for it was discovered thousands of years ago. And human Go players have built up this immense knowledge base over the years. It's studied very deeply and played by something like 50 million players across the world, mostly in China, Japan and Korea, where it's an important part of the culture, so much so that it's considered one of the four ancient arts that was required by Chinese.

Scholars. So there's a deep history there, but there's interesting qualities. So if I compare to chess, chess is in the same way as it is in Chinese culture for Go. And chess in Russia is also considered one of the sacred.

Arts.

Yeah. So if we contrast sort of Go with chess, there's interesting qualities about Go. Maybe you can correct me if I'm wrong, but the, the evaluation of a particular static board is not as reliable. Like, you can't. In chess, you can kind of assign points to the different.

Units.

Yeah. And it's kind of a pretty good measure of who's winning, who's losing. It's not so.

Clear. Yeah. So in the game of Go, you, you find yourself in a situation where both players have played the same number of stones. Actually, captures at strong level of play happen very rarely. Which means that at any moment in the game, you've got the same number of white stones and black stones. And the only thing which differentiates how well you're doing is this intuitive sense of where are the territories ultimately going to form on this board. And if you look at the complexity of a real Go position, you know, it's mind boggling, that kind of question of what will happen in 300 moves from now when you see just a scattering of 20 white and black stones intermingled. And so that challenge is the reason why position evaluation is so hard in Go compared to other games. In addition to that, it has an enormous search space. So there's around 10 to the 170 positions in the game of Go. That's an astronomical number. And that search space is so great that traditional heuristic search methods that were so successful and Things like Deep Blue and chess programs just kind of fall over and.

Go. So at which point did reinforcement learning enter your life, your research life, your way of thinking? We just talked about learning. But reinforcement learning is a very particular kind of learning, one that's both philosophically sort of profound, but also one that's pretty difficult to get to work, if we look back in the earth, at least the early days. So when did that enter your life and how did that work.

Progress? So I had just finished working in the games industry at this startup company and I took a year out to discover for myself exactly which path I wanted to take. I knew I wanted to study intelligence, but I wasn't sure what that meant at that stage. I really didn't feel I had the tools to decide on exactly which path I wanted to follow. So during that year I read a lot, and one of the things I read was Saturn and Barto, the sort of seminal textbook on an introduction to reinforcement learning. And when I read that textbook, I just had this resonating feeling that this is what I understood intelligence to be and this was the path that I felt would be necessary to go down to make progress in AI. So I got in touch with Rich Saturn and asked him if he would be interested in supervising me on a PhD thesis in computergo. And he basically said that if he's still alive, he'd be happy to, but unfortunately he'd been struggling with very serious cancer for some years and he really wasn't confident at that stage that he'd even be around to see the end event. But fortunately that part of the story worked out very happily and I found myself out there in Alberta. They've got a great games group out there with a history of fantastic work in board games, as well as Rich Sutton, the father of rl. So it was the natural place for me to go in some sense to study this question. And the more I looked into it, the more strongly I felt that this wasn't just the path to progress in computergo, but really this was the thing I'd been looking for. This was. Really an opportunity to frame what intelligence means, what are the goals of AI in a clear, single, clear problem definition, such that if we're able to solve that clear, single problem definition, in some sense we've cracked the problem of.

AI. So to you, reinforcement learning ideas.

At least sort of echoes of it would be at the core of intelligence, is at the core of intelligence. And if we ever create a human level intelligence system, it would be at the core of that kind of.

System. Let me say it this way. I think it's helpful to separate out the problem from the solution. So I see the problem of intelligence. I would say it can be formalized as the reinforcement learning problem. And that that formalization is enough to capture most, if not all of the things that we mean by intelligence. That they can all be brought within this framework and gives us a way to access them in a meaningful way that allows us as scientists to understand intelligence and us as computer scientists to build them. And so in that sense, I feel that it gives us a path, maybe not the only path, but a path towards AI. And so do I think that any system in the future that's solved AI would have to have RL within it? Well, I think if you ask that, you're asking about the solution methods. I would say that if we have such a thing, it would be a solution to the RL problem. Now, what particular methods have been used to get there? Well, we should keep an open mind about the best approaches to actually solve any problem. And the things we have right now for reinforcement learning, maybe I believe they've got a lot of legs, but maybe we're missing some things. Maybe there's going to be better ideas. I think we should keep. Let's remain modest. And we're at the early days of this field and there are many amazing discoveries ahead of.

Us. For sure, the specifics, especially of the different kinds of RL approaches Currently there could be other things that fall under the very large umbrella of rl. But if it's okay, can we take a step back and kind of ask the basic question of what is to you, reinforcement.

Learning? So reinforcement learning is the study and the science and the problem of intelligence in the form of an agent that interacts with an environment. So the problem you're trying to solve is represented by some environment like the world in which that agent is situated. And the goal of RL is clear that the agent gets to take actions. Those actions have some effect on the environment. And the environment gives back an observation to the agent saying this is what you see or sense. And one special thing which it gives back is called the reward signal, how well it's doing in the environment. And the reinforcement learning problem is to simply take actions over time so as to maximize that reward.

Signal. So a couple of basic questions. What types of RL approaches are there? So I don't know if there's a nice brief inwards way to paint the picture of value based, model based, policy based reinforcement.

Learning. Yeah. So now if we think about, okay, so there's this ambitious problem definition of rl. It's really truly ambitious. It's trying to capture and encircle all of the things in which an agent interacts with an environment and say, well, how can we formalize and understand what it means to crack that? Now let's think about the solution method. Well, how do you solve a really hard problem like that? Well, one approach you can take is to decompose that very hard problem into pieces that work together to solve that hard problem. And so you can kind of look at the decomposition that's inside the agent's head, if you like, and ask, well, what, what form does that decomposition take? And some of the most common pieces that people use when they're kind of putting the solution method together? Some of the most common pieces that people use are whether or not that solution has a value function that means is it trying to predict, explicitly trying to predict how much reward it will get in the future? Does it have a representation of a policy that means something which is deciding how to pick actions? Is that decision making process explicitly represented? And is there a model in the system? Is there something which is explicitly trying to predict what will happen in the environment? Those three pieces are to me, some of the most common building blocks. I understand the different choices in RL as choices of whether or not to use those building blocks when you're trying to decompose the solution. Should I have a value function represented? Should I have a policy representative? Should I have a model represented? And there are combinations of those pieces and of course, other things that you could add into the picture as well. But those three fundamental choices give rise to some of the branches of RL with which we are very.

Familiar. And so those, as you mentioned, there is a choice of what's specified or modeled explicitly. And the idea is that all of these are somehow implicitly learned within the system. So it's almost a choice of how you approach a problem. Do you see those as fundamental differences or are these almost like small specifics, like the details of how you solve the problem, but they're not fundamentally different from each.

Other. I think the fundamental idea is maybe at the higher level, the fundamental idea is the first step of the decomposition is really to say, well, how are we really going to solve any kind of problem where you're trying to figure out how to take actions? And just from this stream of observations, you've got some agent situated in its sensory motor stream and getting all these observations in getting to take these actions and what should it do? How can you even broach that problem? Maybe the complexity of the world is so great that you can't even imagine how to build a system that would understand how to deal with that. And so the first step of this decomposition is to say, well, you have to learn. The system has to learn for itself. And so note that the reinforcement learning problem doesn't actually stipulate that you have to learn. You could maximize your rewards without learning. It just wouldn't do a very good job of it. So learning is required because it's the only way to achieve good performance in any sufficiently large and complex environment. So that's the first step. And so that step gives commonality to all of the other pieces. Because now you might ask, well, what should you be learning? What does learning even mean in this sense? Learning might mean, well, you're trying to update the parameters of some system, which is then the thing that actually picks the actions. And those parameters could be representing anything. They could be parameterizing a value function or a model or a policy. And so in that sense, there's a lot of commonality in that whatever is being represented, there is the thing which is being learned, and it's being learned with the ultimate goal of maximizing rewards. But the way in which you decompose the problem is really what gives the semantics to the whole system. Are you trying to learn something to predict well, like a value function or a model? Are you learning something to perform well, like a policy and the form of that objective? Like, it's kind of giving the semantics to the system. And so it really is, at the next level down, a fundamental choice. And we have to make those fundamental choices as system designers or enable our algorithms to be able to learn how to make those choices for.

Themselves. So then the next step you mentioned, the very first thing you have to deal with, is can you even take in this huge stream of observations and do anything with it? So the natural next basic question is, what is deep reinforcement learning? And what is this idea of using neural networks to deal with this huge incoming.

Stream? So, amongst all the approaches for reinforcement learning, deep reinforcement learning is one family of solution methods that tries to utilize powerful representations that are offered by neural networks to represent any of these different components of the solution of the agent, like whether it's the value function or the model or the policy. The idea of deep learning is to say, well, here's a powerful toolkit that's so powerful that it's universal in the sense that it can represent any function, and it can Learn any function. And so if we can leverage that use universality, that means that whatever we need to represent for our policy or for our value function or for a model, deep learning can do it. So that deep learning is one approach that offers us a toolkit that has no ceiling to its performance. That as we start to put more resources into the system, more memory and more computation and more data, more experience of more interactions with the environment, that these are systems that can just get better and better and better at doing whatever the job is they've asked them to do. Whatever we've asked that function to represent, it can learn a function that does a better and better job of representing that knowledge. Whether that knowledge be estimating how well you're going to do in the world, the value function, whether it's going to be choosing what to do in the world, the policy, or whether it's understanding the world itself, what's going to happen next, the.

Model. Nevertheless, the fact that neural networks are able to learn incredibly complex representations that allow you to do the policy, the model or the value function is, at least to my mind, exceptionally beautiful and surprising. Was it surprising to you? Can you still believe it works as well as it does? Do you have good intuition about why it works at all and works as well as it.

Does? I think. Let me take two parts to that question. I think it's not surprising to me that the idea of reinforcement learning works because in some sense I feel it's the only thing which can ultimately. And so I feel we have to address it and there must be success is possible because we have examples of intelligence and it must at some level be able to possible to acquire experience and use that experience to do better in a way which is meaningful to environments of the complexity that humans can deal with. It must be. Am I surprised that our current systems can do as well as they can do? I think one of the big surprises for me and a lot of the community is really the fact that deep learning can continue to perform so well, despite the fact that these neural networks that they're representing have these incredibly non linear kind of bumpy surfaces which to our kind of low dimensional intuitions make it feel like surely you're just going to get stuck and learning will get stuck because you won't be able to make any further progress. And yet the big surprise is that learning continues and these what appear to be local optima turn out not to be, because in high dimensions, when we make really big neural nets, there's always a way out and there's a way to go even lower. And then you still not in a local optima because there's some other pathway that will take you out and take you lower still. And so no matter where you are learning can proceed and do better and better and better without bound. And so that is a surprising and beautiful property of neural nets, which I find elegant and beautiful and somewhat shocking that it turns out to be.

The case, as you said, which I really like to our low dimensional intuitions. That's.

Surprising. Yeah, we're very tuned to what working within a three dimensional environment. And so to start to visualize what a billion dimensional neural network surface that you're trying to optimize over what that even looks like is very hard for us. And so I think that really if you try to account for essentially the AI winter where people gave up on neural networks, I think it's really down to that lack of ability to generalize from low dimensions to high dimensions. Because back then we were in the low dimensional case, people could only build neural nets with 50 nodes in them or something. And to imagine that it might be possible to build a billion dimensional neural net and it might have a completely different, qualitatively different property was very hard to anticipate. And I think even now we're starting to build the theory to support that and it's incomplete at the moment, but all of the theory seems to be pointing in the direction that indeed this is an approach which truly is universal, both in its representational capacity, which was known, but also in its learning ability, which is.

Surprising. It makes one wonder what else we're missing due to our low dimensional intuitions that will seem obvious once it's.

Discovered. I often wonder, when we one day do have AIs which are superhuman in their abilities to understand the world, what will they think of the algorithms that we developed back now? Will it be looking back at these days and thinking that will we look back and feel that these algorithms were naive first steps, or will they still be the fundamental ideas which are used even in 100,000, 10,000.

Years? Yeah, they'll watch back to this conversation with a smile, maybe a little bit of a laugh. I mean, my sense is, I think just like when we used to think that the sun revolved around the earth, they'll see our systems of today, reinforcement learning as too complicated, that the answer was simple all along. There's something just like you said in the game of Go. I mean, I love the systems of like cellular automata, that there's simple rules from which incredible Complexity emerges. So it feels like there might be some really simple approaches, just like Rich Sutton says. Right. These simple methods with compute over time seem to prove to be the Most.

Effective. I 100% agree. I think that if we try to anticipate what will generalize well into the future, I think it's likely to be the case that it's the simple, clear ideas which will have the longest legs and which will carry us furthest into the future. Nevertheless, we're in a situation where we need to make things work today, today. And sometimes that requires putting together more complex systems where we don't have the, the full answers yet as to what those minimal ingredients might.

Be. So, speaking of which, if we could take a step back to go, what was Mogo and what was the key idea behind the.

System? So back during my PhD on computergo, around about that time, there was a major new development which actually happened in the context of Computer Go, and, and it was really a revolution in the way that heuristic search was done. The idea was essentially that a position could be evaluated or a state in general could be evaluated, not by humans saying whether that position is good or not, or even humans providing rules as to how you might evaluate it, but instead by allowing the system to randomly play out the game until the end, multiple times, and taking the average of those outcomes as the prediction of what will happen. So, for example, if you're in the game of Go, the intuition is that you take a position and you get the system to kind of play random moves against itself all the way to the end of the game, and you see who wins. And if black ends up winning more of those random games than white, well, you say, hey, this is a position that favors white. And if white ends up winning more of those random games than black, then it favors white. So that idea was known as Monte Carlo Search. And a particular form of Monte Carlo Search that became very effective and was developed in Computer Go first by Remy Coulomb in 2006 and then taken further by others was something called Monte Carlo Tree Search, which basically takes that same idea and uses that insight to evaluate. Every node of a search tree is evaluated by the average of the random playouts from that node onwards. This idea was very powerful and suddenly led to huge leaps forward in the strength of computer GO playing programs. And among those, the strongest of the go playing programs in those days was a program called Mogo, which was the first program to actually reach human master level on small boards, nine by nine boards. And so this was a program by someone called Silvangelli, who's a good colleague of mine, but I worked with him a little bit in those days, part of my PhD thesis. And Mogo was a first step towards the latest successes we saw in Computer go. But it was still missing a key ingredient Mogo was evaluating purely by random rollouts against itself. And in a way, it's truly remarkable that random play should give you anything at all. Why, in this perfectly deterministic game that's very precise and involves these very exact sequences, why is it that randomization is helpful? And so the intuition is that randomization captures something about the, the nature of the search tree from a position, that you're understanding the nature of the search tree from that node onwards by using randomization. And this was a very powerful.

Idea. And I've seen this in other spaces, talk to Richard Karp and so on. Randomized algorithms somehow magically are able to do exceptionally well. And simplifying the problem somehow makes you wonder about the fundamental nature of randomness in our.

Universe. It seems to be a useful.

Thing. So from that moment, can you maybe tell the origin story and The Journey of.

AlphaGo? Yeah. So programs based on Monte Carlo tree search were a first revolution in the sense that they led to suddenly programs that could play the game to any reasonable level, but they plateaued. It seemed that no matter how much effort people put into these techniques, they couldn't exceed the level of amateur Dan level GO players. So strong players, but not anywhere near the level of professionals, never mind the world champion. And so that brings us to the birth of AlphaGo, which happened in the context of a startup company known as DeepMind. I heard of them where a project was born, and the project was really a scientific investigation where myself and Aja Huang and an intern, Chris Madison, were exploring a scientific question. And that scientific question was really, is there another fundamentally different approach to this key question of go? The key challenge of how can you build that intuition in how can you just have a system that could look at a position and understand what move to play or how well you're doing in that position, who's going to win? And so the deep learning revolution had just begun. Systems like imagenet had suddenly been won by deep learning techniques back in 2012. And following that, it was natural to ask, well, if, if deep learning is able to scale up so effectively with images, to understand them enough to classify them, well, why not Go? Why not take the black and white stones of the Go board and build a system which can understand for itself what that means? In terms of what move to pick or who's going to win the game, black or white. And so that was our scientific question, which we were probing and trying to understand. And as we started to look at it, we discovered that we could build a system. So in fact, our very first paper on AlphaGo was actually a pure deep learning system which was trying to answer this question. And we showed that actually a pure deep learning system with no search at all was actually able to reach human Dan level, master level at the full game of Go, 19 by 19 boards. And so without any search at all, suddenly we had systems which were playing at the level of the best Monte Carlo tree search systems, the ones with randomized.

Rollouts. So first of all, sorry to interrupt, but that's kind of a groundbreaking notion. That's like basically a definitive step away from a couple of decades of essentially search dominating AI. So how does that make you feel? Was it surprising from a scientific perspective in general, how to make you.

Feel? I found this to be profoundly surprising. In fact, it was so surprising that we had a bet back then and like many good projects, bets are quite motivating. And the bet was whether it was possible for a system based purely on deep learning, no search at all, to beat a down level human player. And so we had someone who joined our team who was a down level player. He came in and we had this first match against.

Him. Which side of the bet were you on, by the way? The losing or the winning.

Side? I tend to be an optimist with the power of deep learning and reinforcement learning. So the system won and we were able to beat this human down level player. And for me that was the moment where it was like, okay, something special is afoot here. We have a system which without search, is able to already just look at this position and understand things as well as a strong human player. And from that point onwards I really felt that reaching the top levels of human play, professional level, world champion level, I felt it was actually an inevitability and if it was an inevitable outcome, I was rather keen that it would be us that achieved it. So we scaled up. This was something where I had lots of conversations back then with Demis Asabis, the head of DeepMind, who was extremely excited and we made the decision to to scale up the project, brought more people on board and so AlphaGo became something where we had a clear goal which was to try and crack this outstanding challenge of AI to see if we could beat the world's best players. And this led within the space of not so many months to playing against the European champion, Fan Hui, in a match which became memorable in history as the first time a Go program had ever beaten a professional player. And at that time, we had to make a judgment as to whether, when and whether we should go and challenge the world champion. And this was a difficult decision to make. Again, we were basing our predictions on our own progress and had to estimate, based on the rapidity of our own progress when we thought we would exceeds the level of the human world champion. And we tried to make an estimate and set up a match, and that became the AlphaGo vs. Lisa doll match in.

2016. And we should say spoiler alert that AlphaGo was able to defeat Lisa.

Doll. That's right.

Yeah. So maybe we could take even a broader view. AlphaGo involves both learning from expert games and, as far as I remember, a self play component to where it learns by playing against itself. But in your sense, what was the role of learning from expert games there? And in terms of your self evaluation, whether you can take on the world champion, what was the thing that they're trying to do more of, sort of train more on expert games? Or was there now another. I'm asking so many poorly phrased questions, but did you have a hope, a dream, that self play would be the key component at that moment.

Yet? So in the early days of alphago, we used human data to explore the science of what deep learning can achieve. And so when we had our first paper that showed that it was possible to predict the winner of the game, that it was possible to suggest moves, that was done using human.

Data. Oh, solely human.

Data. Yeah. And so the reason that we did it that way was at that time, we were exploring separately the deep learning aspect from the reinforcement learning aspect. That was the part which was new and unknown to me at that time was how far could that be stretched? Once we had that, it then became natural to try and use that same representation and see if we could learn for ourselves using that same representation. And so right from the beginning, actually, our goal had been to build a system using self play. And to us, the human data right from the beginning was an expedient step to help us, for pragmatic reasons, to go faster towards the goals of the project than we might be able to starting solely from self play. And so in those days, we were very aware that we were choosing to use human data, and that might not be the long term holy grail of AI, but that it was something which was extremely useful to us. It helped us to understand the system. It helped us to build deep learning representations which were clear and simple and easy to use. And so really, I would say it served a purpose, not just as part of the algorithm, but something which I continue to use in our research today, which is trying to break down a very hard challenge into pieces which are easier to understand for us as researchers and develop. So if you use a component based on human data, it can help you to understand the system such that then you can build the more principled version later that does it for.

Itself. So, as I said, the AlphaGo victory, and I don't think I'm being sort of romanticizing this notion, I think it's one of the greatest moments in the history of AI. So were you cognizant of this magnitude of the accomplishment at the time? I mean, are you cognizant of it even now? Because to me, I feel like it's something that would. We mentioned what the AGI systems of the future will look back. I think they'll look back at the AlphaGo victory as like, holy crap, they figured it out. This is where. This is where it.

Started. Well, thank you again. I mean, it's funny because I guess I'd been working on computergo for a long time, so I'd been working at the time of the AlphaGo match on ComputerGo for more than a decade. And throughout that decade I'd had this dream of what would it be like really to actually be able to build a system that could play against the world champion. And I imagined that that would be an interesting moment, that maybe some people might care about that and that this might be a nice achievement. But I think when I arrived in Seoul and discovered the legions of journalists that were following us around and the hundred million people that were watching the match online live, I realized that I'd been off in my estimation of how significant this moment was by several orders of magnitude. And so there was definitely an adjustment process to realize that this was something which the world really cared about and which was a watershed moment. And I think there was that moment of realization, which was also a little bit scary, because if you go into something thinking it's going to be maybe of interest and then discover that 100 million people are watching, it suddenly makes you worry about whether some of the decisions you'd made were really the best ones or the wisest or were going to lead to the best outcome. And we knew for sure that there were still imperfections in alphago which were going to be exposed to the whole world. Watching. And so, yeah, it was, I think, a great experience and I feel privileged to have been part of it, privileged to have led that amazing team. I feel privileged to have been in a moment of history, like you say, but also lucky that in a sense, I was insulated from the knowledge of. I think it would have been harder to focus on the research if the full kind of reality of what was going to come to pass had been known to me and the team. I think it was. We were in our bubble and we were working on research and we were trying to answer the scientific questions, and then, bam, the public sees it. And I think it was better that.

Way. In retrospect, were you confident that, I guess, what were the chances that you could get to win? So, just like you said, I'm a little bit more familiar with another accomplishment that we may not even get a chance to talk to. I talked to oriole Vinales about AlphaStar, which is another incredible accomplishment. But here, you know, with Alphastar and beating the Starcraft, there was like, already a track record with Alphago. This is like the really first time you get to see reinforcement learning face the best human in the world. So what was your confidence like? What was the.

Odds? Well, we.

Actually. Was there a.

Bet? Funnily enough, there was. So just before the match, we weren't betting on anything concrete, but we all held out a hand. Everyone in the team held out a hand at the beginning of the match, and the number of fingers that they had out on that hand was supposed to represent how many games they thought we would win against Lisa Dahl. And there was an amazing spread in the team's predictions. But I have to say I predicted 4:1. And the reason was based purely on data. So I'm a scientist first and foremost, and one of the things which we had established was that AlphaGo, in around one in five games would develop something which we called a delusion, which was a kind of hole in its knowledge where it wasn't able to fully understand everything about the position. And that hole in its knowledge would persist for tens of moves throughout the game. And we knew two things. We knew that if there were no delusions, that AlphaGo seemed to be playing at a level that was far beyond any human capabilities. But we also knew that if there were delusions, the opposite was true. And in fact, that's what came to pass. We saw all of those outcomes, and Lisa Dol, in one of the games, played a really beautiful sequence that AlphaGo just hadn't predicted. And after that, it led it into this situation where it was unable to really understand the position fully and found itself in one of these delusions. So indeed, 4:1 was the.

Outcome. So, yeah. And can you maybe speak to it a little bit more? What were the five games like? What happened? Is there interesting things that. That come to memory in terms of the play of the human on the.

Machine? So I remember all of these games vividly. Of course, moments like these don't come too often in the lifetime of a scientist. And the first game was magical because it was the first time that a computer program had defeated a world champion in this grand challenge of go. And there was a moment where AlphaGo invaded Lisa Doll's territory towards the end of the game. And that's quite an audacious thing to do. It's like saying, hey, you thought this was going to be your territory in the game, but I'm going to stick a stone right in the middle of it and prove to you that I can break it up. And Lisa Doll's face just dropped. He wasn't expecting a computer to do something that audacious. The second game became famous for a move known as Move 37. This was a move that was played by AlphaGo that broke all of the conventions of Go that the Go players were so shocked by this, they thought that maybe the operator had made a mistake. They thought that there was something crazy going on. And it just broke every. Every rule that Go players are taught from a very young age. They're just taught this kind of move called a shoulder hit. You can only play it on the third line or the fourth line, and AlphaGo played it on the fifth line, and it turned out to be a brilliant move and made this beautiful pattern in the middle of the board that ended up winning the game. And so this really was a clear instance where we could say computers exhibited creativity, that this was really a move that was something humans hadn't known.