MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL)

Today I'd like to overview the exciting field of deep reinforcement learning. Introduce, overview and provide you some of the basics. I think it's one of the most exciting fields in artificial intelligence. It's marrying the power and the ability of deep neural networks to represent and comprehend the world, with the ability to act on that understanding, on that representation, taking as a whole, that's really what the creation of intelligent beings is, understand the world and act. And the exciting breakthroughs that recently have happened captivate our imagination about what's possible. And that's why this is my favorite area of deep learning and artificial intelligence in general. And I hope you feel the same. So what is deep reinforcement learning? We've talked about deep learning, which is taking samples of data, being able to, in a supervised way, compress, encode the representation of that data in a way that you can reason about it. Now we take that power and apply it to the world where sequential decisions are to be made. So it's looking at problems and formulations of tasks where an agent, an intelligent system, has to make a sequence of decisions and the decisions that are made have an effect on the world around the agent. How, how do all of us, any intelligent being that is tasked with operating in the world, how do you learn anything, especially when you know very little in the beginning? This trial and error is the fundamental process by which reinforcement learning agents learn. And the deep part of deep reinforcement learning is neural networks is using the frameworks. And reinforcement learning, where the neural network is doing the representation of the world based on which the actions are made. And we have to take a step back when we look at the types of learning, sometimes the terminology itself can confuse us to the fundamentals. There is supervised learning, there's semi supervised learning, there's unsupervised learning, there's reinforcement learning. And there's this feeling that supervised learning is really the only one where you have to perform the manual annotation, where you have to do the large scale supervision. That's not the case. Every type of machine learning is supervised learning. It's supervised by a loss function or a function that tells you what's good and what's bad. You know, even looking at our own existence is how we humans figure out what's good and bad. There's all kinds of sources, direct and indirect, by which our morals and ethics, we figure out what's good and bad. The difference between supervised and unsupervised. And reinforcement learning is the source of that supervision. What's implied when you say unsupervised is that the cost of human labor required to attain the supervision is low. But it's never turtles all the way down. It's turtles, and then there's a human at the bottom. At some point, there needs to be human intervention, human input to provide what's good and what's bad. And this will arise in reinforcement learning as well. We have to remember that, because the challenges and the exciting opportunities of reinforcement learning lie in the fact of how do we get that supervision in the most efficient way possible. But supervision nevertheless is required for any system that has an input and an output that's trying to learn like a neural network does. To provide an output that's good, it needs somebody to say what's good and what's bad for you. Curious about that. There's been a few books, a couple written throughout the last few centuries, from Socrates to Nietzsche. I recommend the latter especially. So let's look at supervised learning and reinforcement learning. I'd like to propose a way to think about the difference that is illustrative and useful when we start talking about the techniques. So supervised learning is taking a bunch of examples of data and learning from those examples, where ground truth provides you the compressed semantic meaning of what's in that data. And from those examples, one by one, whether it's sequences or single samples, we learn how to then take future such samples and interpret them. Reinforcement learning is teaching what we teach an agent through experience, not by showing a singular sample of a data set, but by putting them out into the world. The distinction there, the essential element of reinforcement learning, then for us. Now we'll talk about a bunch of algorithms. But the essential design step is to provide the world in which to experience. The agent learns from the world, from the world, it gets the dynamics of that world, the physics of the world, from that world, it gets the rewards, what's good and bad. And us, as designers of that agent, do not just have to do the algorithm, we have to do design the world in which that agent is trying to solve a task. The design of the world is the process of reinforcement learning. The design of examples, the annotation of examples, is the world of supervised learning. And the essential, perhaps the most difficult element of reinforcement learning is the reward, the good versus bad. Here a baby starts walking across the room. We want to define success as a baby walking across the room and reaching the destination, that's success. And failure is the inability to reach that destination. Simple. And reinforcement learning in humans, the way we learn from these very few examples appear to learn from very few Examples through trial and error is a mystery, a beautiful mystery, full of open questions. It could be from the huge amount of data, 230 million years worth of bipedal data that we've been walking mammals, walking, ability to walk or 500 million years, the ability to see, having eyes. So that's the hardware side. Somehow genetically encoded in us is the ability to comprehend this world extremely efficiently. It could be through not the hardware, not the 500 million years, but the few minutes, hours, days, months, maybe even years in the very beginning when we're born. The ability to learn really quickly through observation, to aggregate that information, filter all the junk that you don't need, and be able to learn really quickly through imitation. Learning through observation. The way for walking that might mean observing others to walk. The idea there is, if there was no other around, we would never be able to learn the fundamentals of this walking. Or as efficiently it's through observation. And then it could be the algorithm totally not understood is the algorithm that our brain uses to learn the back propagation that's in artificial neural networks. The same kind of processes not understood in the brain that could be the key. So I want you to think about that as we talk about the very trivial by comparison accomplishments and reinforcement learning and how do we take the next steps. But it nevertheless is exciting to have machines that learn how to act in the world. The process of learning. For those who have fallen in love with artificial intelligence, the process of learning is thought of as intelligence. It's the ability to know very little and through experience, examples, interaction with the world in whatever medium, whether it's data or simulation, so on, be able to form much richer and interesting representations of that world, be able to act in that world. That's the dream. So let's look at this stack of what it means to be an agent in this world. From top, the input to the bottom, the output, there's an environment. We have to sense that environment. We have just a few tools. Us humans have several sensory systems. On cars you can have LiDAR, camera, stereo vision, audio, microphone, networking, GPS, IMU sensors. So on whatever robot you can think about, there's a way to sense that world. And you have this raw sensory data. And then once you have the raw sensory data, you're tasked with representing that data in such a way that you can make sense of it. As opposed to all the raw sensors in the eye, the cones and so on that table, just giant stream of high bandwidth information. We have to be able to form higher abstractions of features based on which we can reason from edges to corners to faces, and so on. That's exactly where deep learning neural networks have stepped in. To be able to, in an automated fashion, with as little human input as possible, be able to form higher order representations of that information. Then there's the learning aspect. Building on top of the greater abstractions formed through representations. Be able to accomplish something useful, whether it's discriminative task, a generative task, and so on, based on the representation. Be able to make sense of the data, be able to generate new data and so on, from sequence to sequence to sequence to sample, from sample to sequence, and so on and so forth to actions as we'll talk about. Then there is the ability to aggregate all the information that's been received in the past to the useful information that's pertinent to the task at hand. It's the thing, the old it looks like a duck, quacks like a duck, swims like a duck. Three different data sets. I'm sure there's state of the art algorithms for the three. Image classification, audio recognition, video classification, activity recognition, so on. Aggregating those three together is still an open problem, and that could be the last piece. Again, I want you to think about as we think about reinforcement learning agents. How do we play? How do we transfer from the game of Atari to the game of Go, to the game of Dota, to the game of a robot navigating an uncertain environment in the real world. And once you have that, once you sense the raw world, once you have a representation of that world, then we need to act, which is provide actions within the constraints of the world in such a way that we believe can get us towards success. The promise. Excitement of deep learning is the part of the stack that converts raw data into meaningful representations. The promise. The dream of deep reinforcement learning is going beyond and building an agent that uses that representation and acts achieve success in the world. That's super exciting. The framework and the formulation of reinforcement learning at its simplest is that there's an environment and there's an agent that acts in that environment. The agent senses the environment by some observation, whether it's partial or complete observation of the environment. And it gives the environment an action, it acts in that environment, and through the action, the environment changes in some way, and then a new observation occurs. And then also as you provide the action, you make the observations, you receive a reward. In most formulations of this framework, this entire system has no memory that the only thing you need to be concerned about is the state you came from. The state you arrived in and the reward received. The open question here is what can't be modeled in this kind of way? Can we model all of it from human life to the game of God? Can all of this be modeled in this way? And what are. Is this a good way to formulate the learning problem of robotic systems in the real world and simulated world? Those are the open questions. The environment could be fully observable or partially observable like in poker. It could be single agent or multi agent. Atari versus driving like deep traffic, deterministic or stochastic. Static versus dynamic. Static is in chess, dynamic again in driving in most real world applications. Discrete versus continuous like games. Chess or continuous and carpool balancing a pole on a cart. The challenge for RL in real world applications is that as a reminder, supervised learning is teaching by example learning by example, teaching. From our perspective, reinforcement learning is teaching by experience. And the way we provide experience to reinforcement learning agents currently, for the most part, is through simulation or through highly constrained real world scenarios. So the challenge is in the fact that most of the successes is with systems environments that are simulatable. So there's two ways to then close this gap. Two directions of research and work. One is to improve the algorithms, improve the ability of the algorithms to form policies that are transferable across all kinds of domains, including the real world, including especially the real world. So train and simulation transfer to the real world. Or as we improve the simulation in such a way that the fidelity of the simulation increase, increases to the point where the gap between reality and simulation is. Is minimal to a degree that things learned in simulation are directly, trivially transferable to the to the real world. Okay, the major components of an RL agent. An agent operates based on a strategy called a policy. It sees the world, it makes a decision that's a policy, makes a decision how to act, sees the reward, sees a new state, acts, sees the reward, sees a new states and acts. And this repeats forever until a terminal state. The value function is the estimate of how good a state is or how good a state action pair is. Meaning taking an action in a particular state. How good is that ability to evaluate that? And then the model different from the environment from the perspective of the agent. So the environment has a model based on which it operates. And then the agent has a representation best understanding of that model. So the purpose for an RL agent in this simply formulated framework is to maximize reward. The way that the reward mathematically and practically is talked about is with a discounted framework. So we discount Further and further future reward. So the reward that's farther into the future is, means less to us in terms of maximization than reward that's in the near term. And so why do we discount it? So first, a lot of it is a math trick to be able to prove certain aspects, analyze certain aspects of convergence and in general, on a more philosophical sense, because environments either are or can be thought of as stochastic random, it's very difficult to. There's a degree of uncertainty which makes it difficult to really estimate the reward there'll be in the future because of the ripple effect of the uncertainty. Let's look at an example. A simple one helps us understand policies, rewards, actions. There's a robot in the room. There's. There's 12 cells in which you can step. It starts in the bottom left. It tries to get rewards. On the top right there's a plus one. It's a really good thing. At the top right it wants to get there by walking around. There's a negative one which is really bad. It wants to avoid that square. And the choice of actions is up, down, left, right, four actions. So you could think of their being a negative reward of 0.04 for each step. So there's a cost to each step and there's a stochastic nature to this world, potentially. We'll talk about both determinists and stochastic. So in the stochastic case, when you choose the action up with an 80% probability, with an 80% chance you move up. But with 10% chance you move left and another 10 move right. So that's stochastic nature. Even though you try to go up, you, you might end up in a blocks to the left and to the right. So for a deterministic world, the optimal policy here, given that we always start in the bottom left, is really shortest path. You can't ever because there's no stochasticity. You're never going to screw up and just fall into the hole in the negative one hole that you just compute the shortest path and walk along that shortest path. Why shortest path? Because every single step hertz there's a negative reward to it. 0.04. So shortest path is the thing that minimizes the reward shortest path to the plus one block. Okay, let's look at a stochastic world. Like I mentioned, the 80% up and then split 10% to left and right. How does the policy change? Well, first of all, we need to have a plan for every single block in the area because you might End up there do due to the stochasticity of the world. Okay, the basic addition there is that we're trying to go avoid up the closer you get to the negative one hole. So just try to avoid up because up. The stochastic nature of up means you might fall into the hole with a 10% chance. And given the.04 step reward, you're willing to take the long way home in some cases, in order to avoid that possibility, the negative one possibility. Now let's look at a reward for each step. If it decreases to negative 2, it really hurts to take every step. Then again, we go to the shortest path, despite the fact that there's a stochastic nature. In fact, you don't really care that you step into the negative one hole, because every step really hurts. You just want to get home. And then you can play with this reward structure, right? Instead of negative 2 or negative 0.04, you can look at negative 0.1, and you can see immediately that the structure of the policy, it changes. So with a higher value, the higher negative reward for each step, immediately the the urgency of the agent increases versus the less urgency, the lower the negative reward. And when the reward flips so it's positive. Every step is a positive. So the entire system, which is actually quite common in reinforcement learning, the entire system is full of positive rewards. And so then the optimal policy becomes the longest path is grad school, Taking as long as possible, never reaching the destination. So what lessons do we draw from robot in the room? Two things. The environment model, the dynamics just there in the trivial example, the stochastic nature, the difference between 80% and 100% and 50%. The model of the world, the environment has a big impact on what the optimal policy is. And the reward structure. Most importantly, the thing we can often control more in our constructs of the task we try to solve in reinforcement learning is the what is good and what is bad and how bad is it and how good is it. Reward structure is a big impact, and that has a complete change, Like Robert Frost said, a complete change on the policy, the choices the agent makes. So when you formulate a reinforcement learning framework as researchers, as students, what you often do is you design the environment, you design the world in which the system learns. Even when your ultimate goal is the physical robot, there's a lot of work still done in simulation. So you design the world, the parameters of that world, and you also design the reward structure. And it can have transformative results. Slight variations in those parameters can be huge results. Huge differences on the policy that's arrived. And of course the example I've shown before I really love is the impact of the changing reward structure might have unintended consequences. And those consequences for real world system can have obviously highly detrimental costs that are more than just a failed game of Atari. So here's a human performing the task, playing the game of coast runners racing around the track. And so it's when you finish first and you finish fast, you get a lot of points. And so it's natural to then, okay, let's do an RL agent and then optimizes for those points. And what you find out in the game is that you also get points by picking up the little green turbo things. And what the agent figures out is that you can actually get a lot more points even by simply focusing on the green turbos. Focusing on the green turbos just rotating over and over, slamming into the wall, fire and everything, just picking it up. Especially because ability to pick up those turbos can avoid the terminal state at the end of finishing the race. In fact, finishing the race means you stop collecting positive reward. So you never want to finish. Collect the turbos. And that's a trivial example. It's not actually easy to find such examples, but they're out there of unintended consequences that can have highly negative detrimental effects when put in the real world. We'll talk about a little bit of robotics. When you put robots, four wheeled ones like autonomous vehicles into the real world and you have objective functions that have to navigate difficult intersections full of pedestrians. So you have to form intent models of those pedestrians. Here you see cars asserting themselves through dense intersections, taking risks. And within those risks that are taken by us humans when we drive vehicles, we have to then encode that ability to take subtle risk into AI based control algorithms perception. Then you have to think about, at the end of the day, there's an objective function. And if that objective function does not anticipate the green turbos that are to be collected and then result in some unintended consequences, could have very negative effects, especially in situations that involve human life. That's the field of AI safety. And some of the folks who talk about DeepMind and OpenAI that are doing incredible work in RL also have groups that are working on AI safety for a very good reason. This is a problem that I believe that artificial intelligence will define some of the most impactful positive things in the 21st century. But I also believe we are nowhere close to solving some of the fundamental problems of AI safety that we also need to address, as we develop those algorithms, examples of reinforcement learning systems. All of it has to do with formulation of rewards, formulation of states and actions. You have the traditional, the often used benchmark of a cart balancing a pole continuous. So the action is the horizontal force of the cart. The goal is to balance the pole so it stays top in the moving cart. And the reward is one at each time step. If the pole is upright and the state measured by the cart, by the agent is the pole angle, angular speed, and of course self sensing of the cart position and the horizontal velocity. Another example here didn't want to include the video because it's really disturbing, but I do want to include the slide because it's really important to think about is by sensing the raw pixels, teaching an agent to play a game of doom. So the goal there is to eliminate all opponents. The state is the raw game pixels, the actions, up, down, shoot, reload, and so on. And the positive reward is when an opponent is eliminated and negative when the agent is eliminated. Simple. I added it here because again on the topic of AI safety, we have to think about objective functions and how that translates into the world of not just autonomous vehicles, but things that even more directly have harm, like autonomous weapon systems. We have a lecture on this in the AGI series and on the robotics platform, the object manipulation, grasping objects. There's a few benchmarks, there's a few interesting applications, learning the problem of grabbing objects, moving objects, manipulating objects, rotating and so on, especially when those objects don't have complicated shapes. And so the goal is to pick up an object in the purely in the grasping object challenge, the state is the visual information. So it's visual, visual based. The raw pixels of the objects. The action is to move the arm, grasp the object, pick it up, and obviously it's positive when the pickup is successful. The reason I'm personally excited by this is because it will finally allow us to solve the problem of the claw which has been torturing me for many years. I don't know, that's not at all why I'm excited by it. Okay. And then we have to think about, as we get greater and greater degree of application in the real world with the robotics like cars, the main focus of my passion in terms of robotics is how do we encode some of the things that us humans encode? How do we, you know, we have to think about our own objective function, our own reward structure, our own model of the environment about which we perceive and reason about, in order to then encode machines that are doing the same. And I believe autonomous driving is in that category. We have to ask questions of ethics, we have to ask questions of risk, value of human life, value of efficiency, money, and so on. All these are fundamental questions that an autonomous vehicle unfortunately has to solve before it becomes fully autonomous. So here are the key takeaways of the real world impact of reinforcement learning agents. On the deep learning side, okay, these neural networks that form higher representation, the fun part is the algorithms. All the different architectures, the different encoder decoder structures, all the attention, self attention, recurrence, LSTMs, GRUs, all the fun architectures and the ability to leverage different data sets in order to discriminate better than perform discriminatory tasks better than, you know, MIT does, better than Stanford, that kind of thing. That's the fun part. The hard part is asking good questions and collecting huge amounts of data that's representative of the task. That's for real world impact, not CBPR publication. Real world impact, a huge amount of data. On the deep reinforcement learning side, the key challenge, the fun part again is the algorithms. How do we learn from data? Some of the stuff I'll talk about today, the hard part is defining the environment, defining the access space and the reward structure. As I mentioned, this is the big challenge and the hardest part is how to crack the gap between simulation and the real world. The leaping lizard, that's the hardest part. We don't even know how to solve that transfer learning problem yet. For the real world impact, the three types of reinforcement learning, there's countless algorithms and there's a lot of ways to taxonomize them. But at the highest level, there's model based and there's model free model based algorithms learn the model of the world. So as you interact with the world, you construct your estimate of how you believe the dynamics of that world operates. The nice thing about doing that is once you have a model or an estimate of a model, you're able to anticipate, you're able to plan into the future. You're able to use the model to, in a branching way, predict how your actions will change the world. So you can plan far into the future. This is the mechanism by which you can do chess in the simplest form, because in chess you don't even need to learn the model, the model is given to you chess, go and so on. The most important way in which they're different, I think, is the sample efficiency is how many examples of data are needed to be able to successfully operate in the world. And, and so model Based methods, because they're constructing a model, if they can, are extremely simple, efficient. Because once you have a model, you can do all kinds of reasoning that doesn't require experiencing every possibility of that model. You can unroll the model to see how the world changes based on your actions. Value based methods are ones that look to estimate the quality of states, the quality of taking a certain action in a certain state. So they're called off policy versus the last category that's on policy. What does it mean to be off policy? It means that they constantly value based agents constantly update how good it is to take an action in a state. And they have this model of that goodness of taking action in a state and they use that to pick them optimal action. They don't directly learn a policy, a strategy of how to act. They learn how good it is to be in a state and use that goodness information to then pick the best one and then every once in a while flip a coin in order to explore. And then policy based methods are ones that directly learn a policy function. So they take as input the world representation of that world with neural networks and as output a action where the action is stochastic. So okay, that's the range of model based, value based and policy based. Here's an image from OpenAI that I really like. I encourage you to, as we further explore here, to look up spinning up in deep reinforcement learning from OpenAI. Here's an image that taxonomizes in the way that I described some of the recent developments in rl. So at the very top, the distinction between model free RL and model based rl. In model free rl, which is what we'll focus on today, there is a distinction between policy optimization so on policy methods and Q learning, which is off policy methods. Policy optimization is methods that directly optimize the policy. They directly learn the policy in some way. And then Q learning off policy methods learn, like I mentioned, the value of taking a certain action in the state. And from that learned that learned Q value be able to choose how to act in the world. So let's look at a few sample representative approaches in this space. Let's start with the one that really was one of the first great breakthroughs from Google, DeepMind. On the deep RL side in solving Atari games, DQN, deep Q learning networks, deep Q networks. And let's take a step back and think about what Q learning is. Q learning looks at the state action value function Q that estimates based on a particular policy or based on an optimal policy. How good Is it to take an action in this state, the estimated reward. If I take an action in this state and continue operating under an optimal policy, it gives you directly a way to say amongst all the actions I have, which action should I take to maximize the reward. Now in the beginning you know nothing. You know you don't have this value estimation, you don't have this Q function. So you have to learn it. And you learn it with the Bellman equation of updating it. You take your current estimate and update it with the reward received. After you take an action here it's off policy and model free. You don't have to have any estimate or knowledge of the world. You don't have to have any policy whatsoever. All you're doing is roaming about the world collecting data. When you took a certain action, here's the reward you received and you're updating gradually this table where the table has stated states on the y axis and actions on the X axis. And the key part there is because you always have an estimate to take an action of the value of taking that action. So you can always take the optimal one. But because you know very little in the beginning that optimal is going to, you have no way of knowing that's good or not. So there's some degree of exploration. The fundamental aspect value based methods, or any RL methods like I said, is trial and error is exploration. So for value based methods like Q learning, the way that's done is with the flip of a coin epsilon greedy. With a flip of a coin you can choose to just take a random action and you slowly decrease epsilon to zero as your agent learns more and more and more. So in the beginning you explore a lot an ε of 1, an ε of 0. In the end, when you're just acting greedy based on the your understanding of the world as represented by the Q value function. For non neural network approaches, this is simply a table. The Q, this Q function is a table like I said, on the Y state X actions and in each cell you, you have a reward that's a discounted reward that you estimate to be received there. And as you walk around with this Bellamy equation, you can update that table, but it's a table nevertheless. Number of states times number of actions. Now if you look at any practical real world problem and an arcade game with raw sensory input is a very crude first step towards the real world. So raw sensory information, this kind of value iteration and updating a table is impractical because here's for a game of breakout, if you look at four consecutive frames of a game of breakout, size of the of the raw sensory input is 84 by 84 pixels. Grayscale, every pixel has 256 values. That's 256 to the power of whatever. 84 times 84 times 4 is whatever it is. It's significantly larger than the number of atoms in the universe. So the size of this Q table, if we use the traditional approach is intractable. Neural networks to the rescue. Deep RL is IRL plus neural networks. Where the neural networks is tasked with taking this in value based methods, taking this Q table and learning a compressed representation of it, learning an approximator for the function from state and action to the value. That's what previously talked about the ability, the powerful ability of neural networks to form representations from extremely high dimensional complex raw sensory information. So it's simple. The framework remains for the most part the same in reinforcement learning. It's just that this Q function for value based methods becomes a neural network and becomes an approximator. Where the hope is, as you navigate the world and you pick up new knowledge through the back propagating the gradient and the loss function that you're able to form a good representation of the optimal Q function. So use neural networks with neural networks are good at which is function approximators and that's dqn. Deep Q network was used to have the initial incredible nice results on arcade games. Where the input is the raw sensory pixels with a few convolutional layers, fully connected layers, and the output is a set of actions, you know, probability of taking that action. And then you sample that and you choose the best action. And so this simple agent with a neural network that estimates that Q function very simple network is able to achieve superhuman performance on many of these arcade games that excited the world because it's taking raw sensory information. The pretty simple network that doesn't in the beginning understand any of the physics of the world, any of the dynamics of the environment. And through that intractable space, the intractable state space is able to learn how to actually do pretty well. The loss function for DQN has two Q functions. One is the expected, the predicted Q value of taking an action in a particular state. And the other is the target against which the loss function is calculated. Which is what is the value that you got once you actually taken that action. And once you've taken that action, the way you calculate the value is by looking at the next step and choosing if you take the best action in the next state, what is going to be The Q function. So there's two estimators going on in terms of neural networks. There's two forward passes here, there's two Q's in this equation. So in traditional dqn, that's just. That's done by a single neural network with a few tricks. And double dqn, that's done by two neural networks. And I mentioned tricks because with this and with most of rl, tricks tell a lot of the story. A lot of what makes systems work is the details in games and robotic systems. In these cases, the two biggest tricks for DQN that will reappear in a lot of value based methods is experience replay. So think of an agent that plays through these games as also collecting memories. You collect this bank of memories that can then be replayed. The power of that. One of the central elements of what makes value based methods attractive is that because you're not directly estimating the policy, but are learning the quality of taking an action in a particular state, you're able to then jump around through your memory and play different aspects of that memory. So learn train the network through the historical data. And then the other trick, simple is like I said, that there is. So the loss function has two cues. So it's a dragon chasing its own tail. It's easy for the loss function to become unstable, so the training does not converge. So the trick of fixing a target network is taking one of the queues and only updating every X steps, every thousand steps and so on. And taking the same kind of network is just fixing it. So for the target network that defines the loss function, just keeping it fixed and only updating regularly. So you're chasing a fixed target with a loss function as opposed to a dynamic one. So you can solve a lot of the Atari games with minimal effort. Come up with some creative solutions here. Break out here. After 10 minutes of training on the left, after 2, 2 hours of training on the right, is coming up with some creative solutions again. It's pretty cool because this is raw pixels, right? We're now like this. Been a few years since this breakthrough. So kind of take it for granted. But I still for the most part captivated by just how beautiful it is that from raw sensory information, neural networks are able to learn to act in a way that actually supersedes humans in terms of creativity, in terms of actual raw performance. It's really exciting. And games of simple form is the cleanest way to demonstrate that. And the same kind of DQN network is able to achieve superhuman performance. And a bunch of different games there's improvements to this like dual dqn. Again the Q function can be decomposed, which is useful into the value estimate of being in that state. And what's called in future slides will be called advantage. So the advantage of taking action in that state, the nice thing of the advantage as a measure is that it's a measure of the action quality relative to the average action that could be taken there. So that's very useful. Advantage versus sort of raw reward is that if all the actions you have to take are pretty good, you want to know well how much better it is in terms of optimization. That's a better measure for choosing actions in a value based sense. So when you have these two estimates, you have these two streams for a neural network and a dualing DQN DGQN where one estimates the value, the other the advantage. And that's again that dueling nature is useful for also when there are many states in which the action is decoupled, the quality of the actions is decoupled from the state. So many states it doesn't matter which action you take. So you don't need to learn all the different complexities, all the topology of different actions when you in a particular state. And another one is prioritize experience replay. Like I said, experience replay is really key to these algorithms. And the thing that syncs some of the policy optimization methods and experience replay is collecting different memories. But if you just sample randomly in those memories, you're now affected. The sampled experiences are really affected by the frequency of those experience occurred, not their importance. So prioritize Experience replay assigns a priority, a value based on the magnitude of the temporal difference learned error. So the stuff you have learned the most from is given higher priority and therefore you get to see through the experience replay process that particular experience more often. Okay, moving on to policy gradients. This is on policy versus Q Learning off policy. Policy gradient is directly optimizing the policy where the input is the raw pixels and the policy network represents the forms of representations of that environment space. And its output produces a stochastic estimate, a probability of the different actions here in the pong the pixels a single output that produces a probability of moving the paddle up. So how to polish gradients? Vanilla policy gradient. Very basic works is you unroll the environment, you play through the environment here pong, moving the paddle up and down and so on, collecting no rewards and only collecting a reward at the very end based on whether you win or lose. Every single action you've taken along the way gets either punished or rewarded based on whether it led to victory or defeat. This also is remarkable that this works at all because the credit assignment there is, I mean every single thing you did along the way is averaged out. It's like muddied. It's the reason that policy gradient methods are more inefficient, but it's still very surprising that it works at all. So the pros versus dqn, the value based methods is that if the world is so messy that you can't learn a Q function. The nice thing about policy gradient, because it's learning the policy directly, that it will at least learn a pretty good policy, usually in many cases faster convergence. It's able to deal with stochastic policies, so value based methods cannot learn stochastic policies. And it's much more naturally able to deal with continuous actions. The cons is it's inefficient versus dqn, it can become highly unstable. As we'll talk about some solutions to this during the training process and the credit assignment. So if we look at the chain of actions that lead to a positive reward, some might be awesome actions, some might be good actions, some might be terrible actions, but that doesn't matter as long as the destination was good. And that's then every single action along the way gets a positive reinforcement. That's the downside. And there's now improvements to that advantage actor critic methods A2C combining the best of value based methods and policy based methods. So having an actor two networks, an actor which is policy based and and that's the one that takes the actions samples the actions from the policy network and the critic that measures how good those actions are. And the critic is value based. So as opposed to in the policy update, the first equation there the reward coming from the destination, the reward being from whether you won the game or not every single step along the way. You now learn a Q value function QSA state in action using the Kritik network. So you're able to now learn about the environment, about evaluating your own actions at every step. So you're much more sample efficient. There's asynchronous from DeepMind and synchronous from OpenAI variants of this but of the actor advantage actor critic framework. But both are highly parallelizable. The difference with a 3C, the asynchronous one is that every single agent so you just throw these agents operating in the environment and they're learning, they're rolling out the games and getting the reward. They're updating the original network asynchronously the global network parameters asynchronously, and as a result they're also operating constantly on outdated versions of that network. The OpenAI approach that fixes this is that there's a coordinator, that there's these rounds where everybody, all the agents in parallel rolling out the episode, but then the coordinator waits for everybody to finish in order to make the update to the global network, and then distributes all the same parameters to all the agents. And so that means that every iteration starts with the same global parameters and that has really nice properties in terms of convergence and stability of the training process. Okay, from Google DeepMind, the deep deterministic policy gradient is combining the ideas of DQN by dealing with continuous action spaces. So taking a policy network, but instead of the actor critic framework, but instead of picking a stochastic policy, having the actor operate in a stochastic nature is picking the best picking a deterministic policy. So it's always choosing the best action, but okay with that. The problem quite naturally is that when the policy is now deterministic, it's able to deal with continuous action space, but because it's deterministic, it's never exploring. So the way we inject exploration into the system is by adding noise. Either adding noise into the action space on the output or adding noise into the parameters of the network that have then that create perturbations in the actions such that the final result is that you try different kinds of things and the scale of the noise, just like with the epsilon greedy in the exploration for dqn, the scale of the noise decreases as you learn more and more. So, on the policy optimization side from OpenAI and others, we'll do a lecture just on this. There's been a lot of exciting work here. The basic idea of optimization on policy optimization with PPO and TRPO is first of all, we want to formulate reinforcement learning as purely an optimization problem. And second of all, if policy optimization the actions you take influences the rest of the optimization process, you have to be very careful about the actions you take. In particular, you have to avoid taking really bad actions when your convergence, the training performance in general collapses. So how do we do that? There's the line search methods, which is where gradient descent or gradient ascent falls under which, which is the. How we train deep neural networks is you first pick a direction of the gradient and then pick the step size. The problem with that is that can get you into trouble here. This is a nice visualization. Walking along a ridge is it can. It can result in you stepping off that ridge. Again, the collapsing of the training process, the performance, the truss region is, is the underlying idea here for the, for the policy optimization methods that first pick the step size so the constraint in various kinds of ways, the, the magnitude of the difference to, to the weights that's applied and then the direction. So placing a much higher priority, not choosing bad actions that can throw you off the optimization path trajectory we should take to that path. And finally on the model based methods and we'll also talk about them in the robotics side. There's a lot of interesting approaches now where deep learning is starting to be used for model based methods when the model has to be learned. But of course when the model doesn't have to be learned, it's given inherent to the game. You know, the model like in Go and chess and so on. AlphaZero has really done incredible stuff. So what's, why is, what is the model here? So the way that a lot of these games are approached. Game of Go, it's turn based. One person goes and another person goes and there's this game tree. At every point there's a set of actions that can be taken and quickly. If you look at that game tree, it grows exponentially. So it becomes huge. Game of Go is the hugest of all in terms of because the number of choices you have is the largest. And there's chess and it gets to checkers and then tic tac toe. And it's just the degree at every step increases, decreases based on the game structure. And so the task for a neural network there is to learn the quality of the board. It's to learn which boards, which game positions are most likely to result in a most useful to explore and result in a highly successful state. So that choice of what's good to explore, what branch is good to go down is where we can have neural networks step in. With AlphaGo it was pre trained. The first success that beat the world champion was pre trained on expert games. Then with AlphaGo Zero it was no pre training on expert systems. So no imitation Learning is just purely through self play through suggesting through playing itself new board positions. Many of these systems use Monte Carlo tree search. And during this search balancing exploitation exploration. So going deep on promising positions based on the estimation of the neural network or with a flip of a coin playing underplayed positions. And so this kind of here you could think of as an intuition of looking at a board and estimating how good that board is and also estimating how good that board is likely to lead to victory down the end. So estimating just general quality and probability of leading to victory, then the next step forward is Alpha 0. Using the same similar architecture with MCTS, Monte Carlo Tree Search, but applying it to different games and applying it and competing against other engines, state of the art engines in Go, in Shogi, in chess, and outperforming them with very few steps. So here's this model based approaches which are really extremely simple, efficient, if you can construct such a model and in robotics, if you can learn such a model can be exceptionally powerful here beating the engines which are far superior to humans already. Stockfish can destroy most humans on earth at the game of chess. The ability through learning, through estimating the quality of a board to be able to defeat these engines is incredible. And the exciting aspect here versus engines that don't use neural networks is that the number it really has to do with based on the neural network, you explore certain positions, you explore certain parts of the tree. And if you look at grandmasters, human players in chess, they seem to explore very few moves. They have a really good neural network at estimating which are the likely branches which will provide value to explore. And on the other side, stockfish and so on are much more brute force in their estimation for the MCTS. And then Alpha 0 is a step towards the grandmaster because the number of branches need to be explored is much, much fewer. A lot of the work is done in the representation formed by the neural network, which is super exciting. And then it's able to outperform Stockfish and chess, it's able to outperform Elmo and Shogi and itself in Go or the previous iterations of AlphaGo Zero and so on. Now the challenge here, the sobering truth, is that majority of real world application of agents that have to act in this world, perceive the world and act in this world, are for the most part not based, have no RL involved. So the action is not learned. You use neural networks to perceive certain aspects of the world, but ultimately the action is not learned from data. That's true for all most of the autonomous vehicle companies or all of the autonomous vehicle companies operating today. And it's true for robotic manipulation, industrial robotics and any of the humanoid robots that navigate in this world under uncertain conditions. All the work from Boston Dynamics doesn't involve any machine learning as far as we know. Now that's beginning to change here with animal the recent development where the certain aspects of the control of robotics is being learned. You're trying to learn more efficient movement, you're trying to learn more robust movement on top of the other controllers. So it's quite exciting through RL to be able to learn some of the control dynamics here that's able to teach this particular robot to be able to get up from arbitrary positions. So it's less hard coding in order to be able to deal with unexpected initial conditions and unexpected perturbations. So it's exciting there in terms of learning the control dynamics and some of the driving policy. So making behavioral driving behavior decisions, changing lanes, turning and so on that if you, if you were here last week heard from Waymo, they're starting to use some RL in terms of the driving policy in order to especially predict the future. They're trying to anticipate intent modeling, predict where the pedestrians, where the cars are going to be based on the environment. They're trying to unroll what's happened recently into the future and beginning to move beyond sort of pure end to end on Nvidia end to end learning approach of the control decisions are actually moving to RL and making long term planning decisions. But again the challenge is the, the gap the leap needed to go from simulation to real world. All most of the work is done from the design of the environment and the design of the reward structure. And because most of that work now is in simulation, we need to either develop better algorithms for training, transfer, learning or close the distance between simulation in the real world. And also we could think outside the box a little bit. At the conversation with Peter Abbeel recently, one of the leading researchers in Deep rl, it kind of on the side quickly mentioned the idea is that we don't need to make simulation more realistic. What we could do is just create an infinite number of simulations or a very large number of simulations. And the naturally the regularization aspect of having all those simulations will make it so that our, our reality is just another sample from those simulations. And so maybe the solution isn't to create higher fidelity simulation or to create transfer learning algorithms. Maybe it's to build a arbitrary number of simulations. So then that step towards creating a agent that work, that works in the real world is a trivial one. And maybe that's exactly whoever created the simulation we're living in and the multiverse that we're living in did next steps. The lecture videos we have several in RL will be made all available on DeepLearning MIT edu. We'll have several tutorials in RL on GitHub. The link is there and I really like the essay from OpenAI on spinning up as a deep RL researcher, you know, if you're interested in getting into research in rl, what are the steps you need to take? From the background of developing the mathematical background, Probstat and multivariate calculus to some of the basics like we covered last week on deep learning, some of the basics ideas in rl, just terminology and so on, some basic concepts, then picking a framework, TensorFlow or PyTorch and learn by doing implement the algorithms I mentioned today, those are the core RL algorithms. So implement algorithms from scratch. It should only take about 200, 300 lines of code. They're actually, when you put it down on paper, quite simple, intuitive algorithms and then read papers about those algorithms that follow after looking not for the big waving performance, the hand waving performance, but for the tricks that were used to train these algorithms. The tricks tell a lot of the story and that's the useful parts that you need to learn and iterate fast on simple benchmark environments. So OpenAI Gym has provided a lot of easy to use environments that you can play with, that you can train an agent in minutes, hours, as opposed to days and weeks. And so iterating fast is the best way to learn these algorithms. And then on the research side, there's three ways to get a best paper award, to publish and to contribute and have an impact in the research community in rl. One is improve an existing approach given a particular benchmarks. There's a few benchmark data sets environments that are emerging. So you want to improve on the existing approach some aspects of the convergence and the performance. You can focus on an unsolved task, there's certain games that just haven't been solved through the RL formulation, or you can come up with a totally new problem that hasn't been addressed by RL before. So with that I'd like to thank you very much. Tomorrow I hope to see you here for deep Traffic. Thanks.