Deep Reinforcement Learning (John Schulman, OpenAI)

So, good morning, everyone. So, I'm gonna talk about some of the core methods in deep reinforcement learning. Um, so the aim of this talk is as follows. Um, first, I'll do a brief introduction to what deep RL is and whether it might make sense to apply it in your problem. Um, I'll talk about some of the core techniques. So, on the one hand, we have the policy gradient methods. Then on the other hand, we have methods that learn a Q function, including Q learning and Sarsa. And I'll talk a little at the end about what are the pros and cons of these different methods. So, first, what is reinforcement learning? It's a branch of machine learning concerned with taking sequences of actions. So often it's described in terms of an agent interacting with a previously unknown environment, and it's trying to maximize some kind of cumulative reward, some kind of reward function that we've defined accumulated over time. And pretty much any kind of task where you have some kind of goal that you want to achieve can be stated in these terms. So this is an extremely general formulation. Uh, what's deep reinforcement learning? It's pretty simple. It's just reinforcement learning where you're using neural networks as function approximators. Um, so the interesting thing about reinforcement learning, in contrast to supervised learning, is, um, it's actually not totally obvious what you should use your neural network to approximate in reinforcement learning. And there are different kinds of algorithms that approximate different things. So one choice is to use the neural network to approximate your policy, which is how the agent chooses its actions. Another choice is to approximate the value functions, which measure how good or bad different states are or how. Or actions. And last, you can use the. You can try to learn a model of the system, a dynamics model, which will make predictions about next states and rewards. Okay, so I'll now give a few examples of different, different places where you might apply reinforcement learning and what the observations and actions would be. So one example is robotics. So here you could imagine a robot where the observations are the camera images and the joint angles of the robot. The actions are the joint torques you're applying, and the reward is going to depend on what you want the robot to do. So this is something we, as the algorithm designer, get to define. So the rewards could be to stay balanced, to navigate to some target location, or something more abstract like serve and protect humans. So reinforcement learning has also been used in a lot of more practical applications. Well, applications that have been practical in the past. I think robotics will be very practical in the future. But for Example one area is inventory management. So this is just one example of how you could use reinforcement learning for a decision making problem. So you have to decide how much to stock up on of every item and your observations would be your current inventory levels. Actions would be how much of each item you're going to purchase and reward is your profit. So people in operations research, this is, this is a subfield, study this kind of problem a lot. Okay. There are also a lot of machine learning problems where people have started to apply reinforcement learning techniques. So one example is attention. So the idea in attention is you don't want to look at the whole input at once. You want to just focus on part of it. So one example of this is with a large image, you might want to just crop out part of it and use that and just do detection on that part of the image. So here your observation would be your current image window action is where to look or where to crop your image and reward is your whether you make a classification error or not. So here the actions are trying to. Here you have to try to choose the right area of the image to look at so you'll do the correct classification. Reinforcement learning has also been used in structured prediction problems which haven't, which in the past often weren't considered to be reinforcement learning problems. But it turns out that like to actually properly solve them, it actually is a reinforcement learning problem. So machine translation, for example. So you get a sentence in the source language and you have to admit a sentence in the target language and you can here your observations are the sentence in the source language, you're omitting one word at a time in the target language and you have some reward function that looks at the whole sentence and tells you how good your translation was. So since this is non differentiable and it's. Yeah, you can't just like differentiate through the whole thing and do gradient descent. So it turns out you have to do, you can use a policy gradient method to optimize your translation system system. So people have started to do that. Okay. So that's just. Those are just a few examples, not exhaustive at all. But I just want to, since I just want to say a little bit about how reinforcement learning fits into the, fits into the picture of all the other types of machine learning problems. So previous, I mean previous courses in this series have talked about supervised learning and unsupervised learning. So how does reinforcement learning relate to them? How is it different? So let's just first compare it to, let's look at supervised learning so in supervised learning, first the environment samples an input output pair from some distribution row. The agent makes a prediction Y hat using its function F, and it receives some loss which tells it if it made the right prediction or the wrong prediction. So the interpretation is environment asks the agent a question and then tells her the right answer. So contextual bandits make this problem a little harder in that they give the learning agent a little bit less information. So now the environment samples an input, but notice that there's not a correct output associated with it. Then the agent takes an action and the agent receives some cost, which is from some probability distribution. So here C is the cost. We're sampling it from some probability distribution, and the agent doesn't know what this probability distribution is. So that's what makes the problem hard. So environment asks the agent a question, the agent answers, and the environment gives her a noisy score on the answer. So this is applied. This actually has a lot of applications. So personalized recommendations is one big one, along with advertising. So you have to decide, like customers who like this. I mean, you have a customer and you know what they liked in the past. So you have to make a prediction about what they're going to like in the future. So you show them appropriate ads or links. Like what either, like what ad, what book you want to try to advertise to them, or what video you want to show them, and so on. So here you can. The big difference between this and the supervised learning setting is you don't have access to the function, the loss function you're trying to optimize. So in particular you can't differentiate through it. We don't know the process that generates C, so we can't compute the grading of the loss function and use that to tune the agent's parameters. So that makes it. So that makes the problem a bit harder. Or you have to use a different kind of algorithm. Lastly, reinforcement learning is almost the same as the contextual bandit setting, except now the environment is stateful. So now instead of sampling the initial state from scratch every time step from the same distribution, the state evolves over time. So you have some transition probability distribution called P here, where the state x, sub t is conditioned on the previous state and the previous action. And that makes the problem quite a bit harder because now, well, for a number of reasons. For one thing, the inputs you're getting depend on what actions you're taking. So now that makes it harder to develop a stable, reliable algorithm, because now as the agent starts to learn, it gets different inputs. So that can lead to all sorts of out of control behavior. And it also means you have delayed effects. Because, because since the system is stateful, you might need to take a lot of actions to get into the right state. So you might need to, you can't just act greedily every time step. You have to, you have to think ahead effectively. Okay, so just to summarize these differences, there are two differences. The first one is you don't have full analytic access to the function you're trying to optimize. You have to query it through interaction. Second, you're interacting with a stateful world, which means that the input you get is going to depend on your previous actions. And if you just take the first of those differences between supervised learning and reinforcement learning, you get the contextual bandit setting. So that's sort of halfway in between. Okay, so I realized that there are multiple. This audience probably has people with different interests. Uh, some people are, um, doing research and want to know about what's the latest in the research world. And some people are, um, want to apply these machine learning techniques to practical applications. Um, so this slide is for the latter group of people. Um, so if you're wondering if you have some problem where you think reinforcement learning might be relevant and you're wondering if you should apply reinforcement learning, um, so, so first I should say that the answer might be no, it might be overkill, especially deep reinforcement learning. So this is a set of fairly new techniques where it's not gonna work out of the box very well. And it's. These techniques aren't that well established. So they require a lot of, they have a lot of knobs to be tuned. So it might be overkill. And yeah, these techniques aren't that well established at the moment. So it might be worth investigating some other methods first. Um, so one, so if your problem has a small number of parameters you're trying to optimize over, and you have a simulator that you can like just do lots of experiments on. Um, then derivative free optimization methods are likely to be better than reinforcement learning, or they're likely to be easier to get working. So these methods just look at, they just, you give them a black box function where you put in a parameter vector and it'll give you a noisy estimate of the score. And these algorithms will just optimize over the parameters of that black box, I mean, that are being put into that black box. So yeah, there's a variety of different methods for derivative free optimization, but these are easier to understand than reinforcement learning and they do kind of work out of the box. Okay. A lot of problems are actually can be seen as contextual bandit problems and the statefulness of the world isn't that relevant. So for example, in advertising, this is where people look at advertising as a contextual bandit problem most of the time because you decide what ad to present the user with and then they either click on it or they don't. But it's really, the user is kind of stateful because if you show them a terrible ad, they might just go and download AdBlock. So there is like, your actions do have some repercussions, but often you can just approximate it as being a contextual bandit problem where there is no state. So there's a better theoretical understanding of contextual bandit problems and methods that have some guarantees. So in that case, so if it is a contextual bandit problem, you might want to use those kind of algorithms instead. And lastly, the operations research field has been using these methods for a while on real problems. And they have a set of methods which are just pretty much the basic algorithms, policy iteration and value iteration. But they're sort of, well, they're well developed ways of doing feature engineering for these problems that end up working pretty decently. So these techniques are also worth considering instead of trying to throw a big neural network at it. Okay, so now, well, now that I've talked about what, why not to use deep reinforcement learning or what it's not good for, I'll just talk about some recent success stories in deep reinforcement learning, which are achievements that probably wouldn't have been possible using these other techniques. So a few years ago there was a pretty influential result by Mani et al. From DeepMind where they used deep Q learning algorithm to play Atari games using the screen images as input. And that's hard because you have these games, you're trying to do different things in all these games. And some of them are kind of complicated. So it's pretty remarkable that you can just use a simple, that a simple algorithm can solve them all. The same algorithm can solve them all. So since then, people have also solved or solved this domain using policy gradients and another algorithm called Dagger. So another big groundbreaking result was beating a champion level player at Go also by DeepMind using a combination of supervised learning from like from expert games plus policy gradients to fine tune the supervised learning policy plus Monte Carlo tree search plus value functions to make the search work better. So a combination of techniques and reinforcement learning robotic. So some of my colleagues at Berkeley had some very nice results learning in real time how to do manipulation Tasks using an algorithm called guided policy search using the PR2 robot. And some of my colleagues and I have been working on robotic locomotion using policy gradient methods. And people have been working on locomotion for a while and have been able to achieve pretty good results using very like highly engineered domain specific methods. But previously there hadn't been much success using general methods to solve it. And last, there have been some Recent results Playing 3D games using policy gradients. In fact, there was even a contest I heard about a couple days ago with this new visdom task, which is pretty nice. So you might want to check out Vizdoom. Okay, so that's, that's it for the high level overview part of this. Um, now I'm gonna start getting into the actual formalism and the technical details. Okay, so the basic object in the field of reinforcement learning is the Markov decision process. Um, so the Markov decision process is defined by the following components. You have a state space. This is all the different states of the system, the action space. These are all the actions the agent can take. And you have this probability distribution which determines the probability of next state and reward. So R is the reward, S, prime is the next state, S and A are the actions. So it's a conditional probability distribution. Sometimes people split this out into a separate reward function, but that's basically an equivalent formulation. Okay. And sometimes there's some extra objects to find. We'll be interested in the. We'll consider an initial state distribution. So this is the world starts out in a certain state, and the typical optimization problem you want to solve given this MDP is to maximize expected cumulative reward. Though there are various ways of defining that more precisely, which I'll go into later. Okay, so there are various different settings of reinforcement learning where you define a slightly different optimization problem. The one we'll be most concerned with is called the episodic setting. So here the agent's experience is split up into a series of episodes which have finite length. So in each episode, we first sample the initial state of the world from some probability distribution mu. And then the agent keeps on acting until the world ends up in some terminal state. So just to give an example of what terminal states might be like and how an episode episodic reinforcement learning problem might look. Um, so one example is when termination is good and you want to terminate the episode as fast as possible. Uh, so if we imagine setting up a task with some kind of taxi robot that should get to the destination as fast as possible, then the episode would be like One trip and it's terminate, it's trying to terminate the episode as fast as possible. Um, another example is a waiter robot where you have a fixed length shift, but the waiter has to accumulate, it has to do as well as possible during that shift. So there the episode has a fixed length. Um, the waiter has to say, maximize tips or customer happiness. Um, and then you could imagine another kind of task where termination is bad and you want the episode to last as long as possible. So you can view life as an example of that. But also you could imagine having a walking robot where you want it to walk as far as possible before it falls over. And in this setting, it's pretty easy to define what the goal is. We just want to maximize the expectation of the total reward per episode. Okay, and the last object we're gonna introduce here is a policy. So the policy is just the function that the agent uses to choose its actions. So we have deterministic policies which are just. The policy is denoted by PI. So we have the action is just some function of the state. And we also have stochastic policies where the policy is a conditional probability distribution. So here is just. We're just gonna make a little bit more precise the setting of the episodic mdp. So first we sample the initial state from this distribution mu. Then we get, we sample the first action from the policy, A zero from the policy. Then we sample next state and reward from the transition probability distribution, and so on, until we reach a terminal state S sub t. And then the quantity we care about is the sum of all these rewards. R0 plus R1 plus R sub T minus 1. And we want to maximize. Yeah, so eta of PI is just defined as the expected total reward of the policy PI. Here's the picture that illustrates exactly the same thing. So you can look at it as a graphical model. Okay? And lastly, in the policy gradient section, in particular, we're gonna be interested in parameterized policies. So here we have a parameter vector theta, which specifies, which specifies exactly what the policy is. So, for example, the family of policies could be just a neural net. You have a certain neural network architecture. And theta specifies all the weights of this neural network. So we could have a deterministic policy, of course, or stochastic policy. And if you're wondering, like, concretely, what would a policy look like? I mean, how do you use a neural network to represent your policy? It's actually exactly. You do exactly the same thing you would do if this were a classification or a regression problem. So in so s, here the state here is your input and the action is your output. So if you have a discrete action space, a discrete set of actions, then you would use a network that outputs a vector of probabilities, the probabilities of the different actions. This is exactly like a classifier. And if you have a continuous action space, you, you would have your neural network output the mean and the diagonal of a covariance matrix of a Gaussian distribution. So this is just like you're doing regression. So you can use the same kind of architectures you'd use in supervised learning. Okay, so that's, that's just the, that's it for the formalism of MDPs. So now I'm going to go into policy gradient methods, which are one broad and general class of reinforcement learning methods which are quite effective. So to give a brief overview of this, here's the intuition of what policy gradient methods are going to do. So here capital R means the sum of rewards of the whole episode. So our optimization problem is we want to maximize the expectation of the total reward given our parameterized policy PI sub theta. And the intuition of how our algorithm is gonna work is we, we're gonna collect a bunch of trajectories. I mean, this is just run a bunch of episodes using our policy. And then we wanna make the good trajectories more probable. So I mean, some of the trajectories were lucky and they were really good. So some of them, the agent was unlucky and they were bad, and the ones that were good, meaning there was high reward. That means the agent probably took good actions there. So we want to increase the probability of the actions from those trajectories. So the most basic version of policy gradient methods just try to make the good trajectories more probable without trying to figure out which were the good actions and which were the bad actions. Slightly better methods or more elaborate methods, try to figure out which actions were good and which ones were bad, and then they try to make the good actions more probable. And lastly, there's another class of methods which actually try to push the actions towards better actions. So they differentiate the loss function with respect to the actions and they try to push the actions to better actions. So we're mostly going to talk about one and two here. Oh, there's a question. Oh yeah, good question. So while we're maximizing over the policy, we're trying to find the best policy. But here the policy is assumed to be parameterized, so there's some parameter vector theta that specifies the policy and now we just want to maximize with respect to theta. Any other questions? Okay, so there's a very, a very fundamental concept which is called the score function gradient estimator, which underlies policy gradient methods. So actually to introduce this, we're not gonna talk about policies in RL at all. We're just gonna assume we have some expectation. We have expectation of F of X where X is sampled from some parameterized probability distribution. So we wanna compute the grading of this expectation with respect to theta. So there's a general formula that'll do this. And the way you derive it is you just write the expectation as an integral and then you just move some things around, you swap the integral with the derivative and you turn it back into an expectation. And what you get at the end is this bottom line which says that you take the expectation of function value times grad log probability. So this is an unbiased estimator of the gradient, meaning if we get enough samples, it'll converge on the right thing. So the way you can compute this estimator, meaning the way you can get a noisy estimate of the grading of the expectation, is you just collect one. You just get one sample from this distribution and then you compute. Then you multiply F of x times grad log probability. So the only requirement for being able to use this estimator is we need to be able to compute the probability density. I mean, we need to be able to analytically compute it and we need to be able to differentiate it with respect to theta. And often it needs to be differentiable. There's another way of deriving it using importance sampling. So you write down the importance sampling estimator for the expectation and then you just swap the derivative with the expectation and you get the same thing. Okay, so. So now let me just give a little bit of intuition about this estimator. Oops. Okay, so F of X is measuring how good the sample x is. So that means that. So g hat here is our gradient estimator, meaning this is what we get if we take one sample x sub I and we compute our estimator. This is our estimate of the gradient. So if we move in direction g hat, that pushes up the log probability of our sample x sub I in proportion to how good it is. So if we have really good, if we got a really good function value, then we're going to try to push up its log probability a lot. And if it was a bad function value, then we're not going to try to push it up very much. So it's Pretty simple intuition. Um, the really nice thing is, um, this is valid even if F of X is discontinuous or if F of X is unknown, meaning you only, you don't get to differentiate it, you just get to see the function values, um, or the sample space is a discrete set. So X doesn't even have to be continuous. And this is quite, this is quite remarkable actually, that you don't even need to have access to the full, you don't need to know exactly what the function is that you're optimizing. You just have to be able to query it for the function value. And this means, this is a way of being able to differentiate functions through a system that has non differentiable pieces. Um, so for example, in, in robotic locomotion, one issue is that you have contacts between the robot's foot and the ground and contact you make and break contact. And that causes a discontinuous change in the dynamics. So that makes it really hard to do smooth optimization techniques to come up with the right behavior. So when you use this kind of gradient estimator along with policy gradients, which I'm gonna talk about very soon, you can actually just differentiate, you can optimize this system even though it has differentiable pieces in it. Okay, so, okay, so here's another little picture of what's going on. So we have our function F of X, which we're trying to maximize the expectation of. And then we have our probability density P of X. Um, so we just sample a bunch of values from our probability density. Those are the blue dots on the x axis. And then we, so then we, we look at the function values and we're trying to push the probability distribution so that the probability goes up at these samples in proportion to the function value. So over on the right side of the curve, that means we're trying to push that probability value up really hard. And on the left side we're pushing it up softly. So what's gonna happen is the probability density is gonna slide to the right. If you can imagine a sort of physical analogy there. Okay, so that's, that's the score function gradient estimator. This is a general technique. It can be used in various machine learning problems. Um, now we're gonna apply it to the reinforcement learning setting. And um, we're gonna take our random variable X to be a whole trajectory. Um, so the trajectory consists of state action, reward, state action, reward, and so on until the end of the episode. And uh, now to get our gradient estimator to get the, to get the gradient of the expected reward. All we've got to do is compute the grad log probability times the total reward. So this probability of the trajectory, that sounds like a really unfriendly quantity because there's a long complicated process that generates this trajectory with lots of time steps, but log, okay, so we can write out what this process is, what this probability density is. So we have, it's just a product of probabilities. We've got our initial, we've got our MU of S0, which is just our initial state distribution. And then every time step we have, we sample the action according to PI and we sample the next state and reward according to our dynamics model. So log turns that product into a sum. And here's the cool part, everything that doesn't contain theta drops out. So the thing is, we didn't know there are parts of this probability distribution P of tau given theta that we don't have access to. So if this is reinforcement learning, we don't assume that we know the dynamics model of the system. We just find out about it by sampling, by doing episodes. So since this product turns into a sum, all the pieces like the log p there and the log mu, which we don't know, just drop out. So it doesn't matter. And what we get in the end is we get a sum of log probabilities of actions. So grad log PI of action given state. So our formula looks like our formula for the gradient of the expectation is just the expectation over trajectories of total reward of the trajectory times grad grad of the sum of all the log probs. So the interpretation of this is, um, we're taking our good trajectories and we're trying to increase their probability in proportion to how good they are. Um, and you can think of this as being similar to supervised learning, where we treat the good trajectories with high rewards as positive examples in our supervised learning problem. So we're using those to train the policy on what, which actions are good. We're basically treating those actions as positive examples. Okay, now we can improve this formula a little bit. Um, so that was just the most basic. I mean, this is an unbiased estimator for the policy gradient. So if we just take that expression inside the expectation on the right hand side and we take one sample of that, it has the right mean. So if we just get enough of them, we're going to get the policy gradient. Um, okay, so that's. But we can also write down some other formulas that have the same mean but have lower variance. So we can come up with better estimators for the policy gradient. Um, and that's actually quite important because the one from the previous slide is really bad when you have a long, a large number of time steps, meaning it has really high variance. So the first thing we can do is you can, we can use the temporal structure of the problem and um, by the way, to derive these next bunch of formulas. It just takes a bunch of really straightforward manipulation where you move around expectations. Um, and I'm not gonna go through all the math, but I'll just say what the formulas are. So, okay, so we can repeat the same argument from the previous slide to just derive the grading estimator for a single reward term. So we end up with that reward term times the grad sum of log probs. And just summing over that we get a new formula where we're not multiplying the sum of the grad log prob of the whole thing times the sum of all rewards. Now so let's look at that bottom formula. Now we have a sum over time of grad log probability of the action that time times the sum of future rewards. Um, so, so now I mean in the formula from the previous slide we would have had all the rewards in that sum. Um, but now we just have the future rewards. And that kind of makes sense because an action can't affect the probability of the previous rewards. Uh, so to figure out if the action is good, we should have only, we should only be looking at the future rewards. So this is a slightly better formula than the one on the previous slide, meaning it has the exact same mean, except different. The expression inside the expectation there has lower variance and we can further reduce the variance by introducing a baseline. So now we can take any old function B which takes in a state and it outputs a real number and we can subtract it from our sum of future rewards. And we didn't affect the mean of the estimator at all. So we, yeah, we didn't change the expectation at all by introducing this baseline. So yeah, for any choice of B, this gives us an unbiased estimator. By the way, if you're not that familiar with the terminology of estimators, what I'm saying is we have a expectation on the right hand side of that formula and the quantity inside that expectation is what's called the estimator. And if we get a bunch of samples, then we can get an estimate of the thing on the left hand side which is what we care about. So when I say it's an unbiased estimator, that just means that, well, that just means that this equation is correct, meaning that the thing on the right hand side equals the thing on the left hand side. So yeah, this works for any choice of baseline. And a near optimal choice is to use the expected return. So the expected sum of future rewards. And the interpretation of that is if we took an action, we only want to increase the probability of the action if it was a good action. Um, so how do we tell if it was a good action? Well, the sum of rewards after that action should have been better than expected. Um, so the B of S is the expected sum of rewards and we're just taking the difference between the measured thing and the expected thing. Yeah. Okay, so that's okay, that's the, that that was a pretty key thing for variance reduction. Um, and I'm gonna introduce one last variance reduction technique. And actually all three of these are really important. So basically nothing's gonna work except for maybe really small scale problems unless you do these things. Um, so the last variance reduction technique is to, to use discounts. Um, so the discount factor ignores delayed effects between actions and rewards. So what we we're gonna do here looks kind of like a hack, but there's an explanation for it which is instead of taking the sum of rewards, we're gonna take a discounted sum of rewards, meaning that we, we add this exponential factor gamma so that when, so when we're multiplying the grad log probability by some future reward, we multiply it by some quantity that decays with time. So people typically use like gamma equals 0.99 or gamma equals 0.95. Uh, so that means like if you use 0.99, that means after 100 time steps you're gonna be, you're gonna be reducing the reward by a factor of one over E. So, so you're exponentially, you're decaying the effect of the future rewards. And the intuition is that an action, the action, shouldn't affect rewards really far in the future like the system should. The sis. The si. Like the assumption is that the system doesn't have really long term memory and it sort of resets or the, there aren't the effects aren't that far delayed. So you can just ignore the interaction between action and a reward way in the future. That's the intuition. So now instead of taking the baseline to be the expected sum of future rewards, we want to do a discounted sum. So now we're measuring if the action was better than expected according to this, like according to the discounted sum. Um, and now there's a more general class of formulas that looks like the one that I just wrote. So this, this one that's on the top of the slide is pretty good. And this is like almost as good as anything you're gonna do to within a small constant factor. Uh, but there's, there's a more general class of formulas that look like grad log probability times some quantity, a hat which we call the advantage estimate. And this is in general just going to be an estimate of. This is an. It has a more precise definition, which is how much, how like how much was this action better than the average action taken by the policy? But in, but informally this just means how much better was the action than expected. So, and, and this formula makes a lot of sense because we want to increase the probability of the good actions and decre. Decrease the probability of the bad ones. So we should, we should increase it in proportion to the goodness of the action. Okay, so just to summarize, so I just told you there's this gradient estimator, meaning there's this expression you can compute which gives you a noisy estimate of the policy gradient. So how do you actually turn this into an algorithm? So this is silly. Algorithm 7. So here's what the algorithm looks like. It's pretty much what you'd expect. You take your policy, you initialize your policy parameter and your baseline function. For each iteration, you execute the, um, the current policy to get a bunch of whole episodes, meaning whole trajectories. And each time step in the, each trajectory you should compute the return, meaning the sum of rewards following that time step, the sum of discounted rewards, and the advantage estimate, which is the sum of discounted rewards minus the baseline. Uh, then you refit the baseline by trying to, um, make the baseline function equal the returns, and then you update the policy using a policy gradient estimator. So you're just doing SGD while updating the baseline as you go along. Yeah, so that's the vanilla policy gradient algorithm. And this is. I'll briefly talk. This has been used to obtain some pretty good results. So it's not that bad of an algorithm, but there, there are several different directions that it can be improved. So one, one issue that you run into is with step sizes. So in supervised learning, step sizes aren't that big of a deal because maybe you take too big of a step. But that's okay, you'll fix it the next update and your current function, your current classifier, for example, doesn't affect what inputs you're getting. So even if you just are doing really, even if your network is just kind of thrashing around for a while because you're taking too big steps, that's not going to cause any problems. But yeah, and reinforce. So step sizes aren't that big of a deal. You can just anneal them. You can start off with a large step size and anneal them down to zero. And that works pretty well. Um, in reinforcement learning, if you take too big of a step, you might wreck your policy. Um, and even if you don't actually change the network that much so you don't lose all your nice features, um, you, you might just change its behavior a little too much and now it's gonna do something totally different and visit a totally different part of state space. Um, so since in reinforcement learning the system is stateful and your state distribution depends on your policy, that makes that like brings a really a different problem. And now like after you took that step, the next batch of data you're gonna get was collected by the bad policy and now you're never gonna recover. Cause you just forgot everything. Yeah. So one way that my colleagues and I, well, one way to fix this is to try to, to try to stop the. Basically try to stop the policy from taking too big of a step. So you can look at the KL divergence between the old policy and the new policy.