Stanford CS230 | Autumn 2025 | Lecture 5: Deep Reinforcement Learning
Lecture 5 (CS230, Stanford) introduces deep reinforcement learning, motivating it with Atari, AlphaGo, StarCraft/Dota and modern language-model alignment successes. The lecture defines RL vocabulary (agent, environment, state, observation, action, reward, return, discount, policy, Bellman equation), walks a toy Q‑learning example (Q table, discounted return, Bellman optimality), extends to deep Q‑networks with pixels (Breakout), and covers practical training techniques (preprocessing, replay memory, epsilon-greedy exploration, target estimation). It closes with advanced topics (PPO, self-play, continuous control) and an introduction to reinforcement learning from human feedback (SFT → reward model → RLHF).
Chapters
Welcome to our fifth lecture in person for Stanford deep learning CS230. Um, today's lecture is going to be about deep reinforcement learning. I actually switched the original plan of talking about neural network interpretability and LLM visualization simply because you, you haven't had the chance to study attention maps, you know, convolution neural networks. And so it would have been an overkill to do that week five. So we're going to talk about neural network interpretability and visualization in a later lecture, actually. But today our focus will be on deep reinforcement learning, which is probably my favorite lecture of the class. I, I feel like I say that every week, but it's okay. I like it. The agenda is pretty packed. We're gonna start with deep reinforcement learning, which you can think of as the marriage between deep learning and reinforcement learning together. The baby is called deep reinforcement learning. And we're going to see how reinforcement learning works and how neural networks can play a part in building reinforcement learning agents. Um, in the second half of the class, we will focus on a very specific.
You know, concept called reinforcement learning from human feedback that you might have heard of. It's one of the core concepts that really made the difference between what you might have remembered as GPT2 and ChatGPT. You know, that's the leap. That's really the technique that has democratized access to LLM because of the performance improvements and the alignment with humans. So we're going to see what, what is this concept of RLHF and how does it work and why does it allow us to align a language model to human preferences?
Ready to go. As always, let's try to make it interactive. Uh, so the motivation behind deep reinforcement learning, and as usual, you're gonna have all the most important papers that are covered in the class listed at the bottom of each slide. Um, reinforcement learning has grown in popularity. Um, one of the, you know, very popular papers called human level control through deep reinforcement learning is. The work from DeepMind has showed us that a single algorithm slash training method can allow us to train AI that can play many, many Atari games better than humans. Single algorithm, over 40, 50 games where it exceeds human capability, which is quite impressive when you thought about the fact that, you know, machine learning used to be niche, and you would have to train a really niche algorithm to perform different tasks. Here's an algorithm that can just learn sort of every Atari game. A little later, you might have heard of AlphaGo. AlphaGo is, is. Is an algorithm that was developed to beat and exceed human performance in The Game of Go, we'll talk little more. The Game of Go is a very complex game, some would argue way more complex than chess from a decision making standpoint and from the possibilities that can happen on the board. And so it actually got solved in 2017 again by the DeepMind team and David Silver's lab later on. And again, another great paper from DeepMind had showed us that reinforcement learning can also be used for strategy game that might be a touch more complex than chess or Go, that might actually involve multiple players playing with each other or against each other. Some of you might have played starcraft, for example. That's an example of a game where it requires a lot of long term thinking, short term thinking. Another one is Dota. Some of you might have played Dota or League of Legend where you have a team playing against another team. Those are examples of games that involve multiple agents playing collaboratively. And it's pretty hard to develop systems that can play with each other against multiple opponents. And finally, most recently, this is 2022. So alongside the release of ChatGPT, this paper that introduces the concept of reinforcement learning with human feedback applied to aligning language models with human preferences. And we talk about that later. So all, all these to say that reinforcement learning allowed us to exceed human performance in a variety of tasks. The first one I want us to think about is the. The Game of Go. So let's say that you were asked to solve the game of Go with classic supervised learning. Okay. Everything we've seen together so far labeled data. How would you solve the game of Go with classic supervised learning?
What data would you collect, what would be the label, etc.
Yes.
Okay, good point. You look at history of plenty of games, hopefully from good players. If you want the algorithm to work and you look at X as the input being the current state of the board, and Y as the next state of the board. And this would tell you what move was selected and you learn the move, essentially. And hopefully if you do that across many, many games, you know, you might, you might see the agent become more attuned to the game and develop a better strategies. So, you know, really, hopefully it's a professional player. What are the disadvantages of that or the shortcomings that you can anticipate?
Yes.
Yeah, yeah, great point. You might not see the entire space of possible states of the board, which is what you said. So you might miss out on a lot of different strategies. So the Game of Go is actually a game with two players, one player that uses the black stones and one Player that uses the white stones and iteratively they're going to place those stones on The Grid, a 13 by 13 grid that you can see on screen with the goal of surrounding their opponents. So you're constantly trying to surround the stones of the opponent and the opponent is trying to surround your stones. And so you can imagine that for every intersection on the grid there is multiple possibilities. Either there's a black stone or a white stone or nothing. And on a 13 by 13 grid, you can imagine how many possibilities of a board state there are. It's impossible to capture all of that with historical moves from professional players. You will just never cover that. The same thing could be said in chess as well. You, you know that even the professional players can plan X number of steps in advanced, but nobody knows where the game takes you. And in the late stages of the games or the end games, players always find themselves playing a different game. And that's part of the magic of being good at chess. So yeah, that's the problem. What's another problem or shortcoming beyond the fact that we can't observe possibly all the states.
Yes.
What that actually lead to in the future that might not make the best decision.
Correct. Correct you if I repeat what you said? Well, first, you don't even know if this was a good move, you know, so maybe it was not even a good move and you're learning something that was not a good move and you're labeling it as a good move. And second, you're actually only getting partial information, meaning you don't have the information of what's in the person's mind and what strategy they're trying to execute. So your store, you're sort of looking at a single example among long term strategy and you can't expect the model to guess what's the long term strategy because it was just trained on X and Y and matching the inputs to a possible output. So you, you don't really have any concept of a strategy at that point. It looks one off at every decisions of the model. Okay, those are really good points. Um, the other one is the ground truth might be ill defined. What I mean by that is.
Even the best humans in the world do not play their best game every day. And even their best game is not the ground truth. And that creates an issue because you're essentially training against a target that is off by a certain margin. You're never gonna get better than the best human. And the best human is not the best possible existing the best possible strategy at every point. So you could argue, what if we get a panel of experts that we're monitoring and those are the best players in the world. Even with a panel of expert that decides every move, you still have an ill defined ground truth, you know.
So that's a big issue. Too many states in the game you mentioned and we will likely not generalize, which is what you said, meaning we're looking at one off situations, we're not looking at entire strategies. And so when we face a board state that we've never seen before because the model was not trained on strategy, it sort of. We get stuck. Yeah.
Okay. And this is an example of a perfect application for reinforcement learning because reinforcement learning is all about delayed labels and making sequences of good decisions. So if you have to remember in one sentence what's rl? RL is making good sequences of decisions. Sequences of good decisions. Sorry.
And do that automatically.
Another way to look at it is the difference between, you know, classic supervised learning and RL is in. In classic supervised learning, you teach by example. In reinforcement learning, you teach by experience, which is also a different concept. You're not just showing cats and non cats to a model. You're actually letting the model experience an environment until it figures out what were the best decision it made and learns from them.
Some examples of reinforcement learning applications, I'm gonna mention them. We, we have, we have gaming, of course, that we already covered.
We.
What are other applications of AI where we need good sequences of decisions?
Yes. Autonomous driving. Yeah, Correct. I mean in driving you could argue RL could work and there's some RL going on. But what you mean, I think is you're, you have some sort of a dynamic planning algorithm that allows you to strategize. If you see a, a red light ahead, you might start slowing down over time, but maybe it will turn green. So you might not slow down completely. This is an example of a strategy that you need, of course. Yeah. Robotic controlling. That's a great example. Also related to autonomous driving. But imagine you want to teach to a robot to move from point A to point B. The number of good decisions that the robot needs to make in terms of moving each of their joints is tremendous. Like, it's actually super unlikely that a robot would move from A to B if it's not trained to make good sequences of decisions.
What else?
The biggest one nobody mentioned yet.
It's not a great application. I don't like it. But it happens to be the biggest one over enforcement learning.
Yeah.
Yeah, yeah. Advertisements. Yeah. Marketing. You're right. So, yeah. We talked about robotics. Advertisement is another example. Advertisements is a long game. Like companies are showing you multiple ads before you buy. And in fact, the reason reinforcement learning is important is because, you know, they're planning a strategy that might lead a buyer to execute a purchase over time. And it requires a long term thinking. So there's a lot of reinforcement learning applied to marketing, advertisement, real time bidding processes, et cetera.
Okay, clear on what RL is and how it differs from classic supervised learning?
No.
Okay, um, so let's put some vocabulary around that concept. In reinforcement learning, you have an agent and the agent interacts with an environment.
As the agent interacts with the environment, the agent will perform certain actions that we will denote a T where T is a time step and the environment will show you states that transition from time step T to time step T plus 1. So subject to an action at an environment may transition from ST to ST plus 1. You can think of the game of Go. I take the action of putting my Blackstone on a certain grid intersection and the environment has changed. It moved from the state has changed. It moved from state time step T to time step T plus 1 where my stone is on the grid. After that state update happens, there's two things that the agent observes. The the the agent observes an observation that we will note OT and a reward rt.
Okay, so those are the vocabulary words. And of course the goal of the agent will be to maximize the rewards.
One thing to know about the observation, we'll talk about it a little more. Um, the observation sometimes is equal to the states. Can someone guess why we might need two concepts instead of a single concept? Why is it important to have a state and an observation?
Yes.
Yes, correct. So in some cases the environment may not be fully.
You know, transparent to the user. And so for example, in chess or in Google, the observation is actually equal to the state. You see everything on your board. All the information is available to you. If you play League of Legends or Starcraft, you know the concept of, you know, I think in English it's called like a cloud or a fog. I think it's a fog. You only see certain part of the map until you have explored everything or until your friends are sort of visiting the other parts of the map. And so the observation is actually, actually less information than the states of the environment. Okay. And then the last piece of vocabulary is a transition. When I refer to a transition, I refer of the process of getting from state T to state t plus one, which means we're in state T. The agent takes an action at it observes OT and a reward RT and it transition to the next state ST plus one. Question.
Wait, what do you mean? You mean is there. Are there examples of environment where the state is so large that the.
Yeah, possibly for computational reasons. Yeah, yeah, you might have games. I mean, look at open world games. Like truly, you could argue, I don't know, there are some games where you might press start and you see the entire environment. But who cares of what's happening 20,000 kilometers west of you if you're in a certain location that might not influence your strategy. So you might actually put some sort of a trust circle or some sort of a circle in which you observe, which you think has 99% of the information you need, possibly for computational reasons. That's a good point. Okay, let's get to a practical example of a reinforcement learning algorithm and develop it together. Uh, this example is called recycling is good because recycling is good, but also because it's a simple example illustrative of reinforcement learning. So let's say we have a small environment with five states. There is a starting state marked in brown, which is state two. It's our. It's our initial state. And then on the right side, sorry, on the left side, you have state one, which is a garbage. And it's great to get to the garbage because you're going to be able to recite to. To put in the garbage the, you know, the stuff that you have in your hands. You know, you're trying to throw away some garbage and the garbage kind of happens to be there. And so we would expect there could be a reward on the other side. If you actually go to the right, you might pass by State 3, which is empty. You might pass by State 4, where there is a chocolate packaging that is left on the ground that you can pick up, and it's good to pick it up. And then on stage five, state five, you have the recycle bean, which is more valuable than the garbage can because you can recycle and you should get better rewards for that. So that's our game. In this game, we define a reward that is associated with the type of behaviors that we want the agents to learn. Um, and the reward is as follow. That's just one example. Plus two for throwing your garbage in the normal can, plus one for picking up the chocolate packaging, and plus 10 if you manage to make it to the recycled bin. Is it clear.
Now the goal will be. And that's the case in reinforcement learning. Oftentimes to maximize the return, we define Formally the return. But think about it as maximize the amount of rewards that you get as you go through this journey and you make your decisions. In this specific game. We have five states and there's three types of state in brown is the initial states. In, we have normal states and we have in blue terminal states. When you get to a terminal state in reinforcement learning, it will typically end the game. It will end one episode of the game. We move to another episode. You'll get back to the starting state or initial state and you'll redo another episode.
The possible actions for agent here are going to be fairly simple left and right.
And we are gonna add an additional rule that is important, which is that the garbage collector comes in three minutes and it takes a minute to get from one state to the other. Why is that an important rule to add to the game?
I guess.
Yeah.
Yeah. Otherwise you just go back and forth between stage three and stage four. You just collect a bunch of chocolate packaging and you never make it to the bin. And so it's not what we want.
Okay, so how do we define the long term return? The long term return is going to be defined as capital R, which is the sum of rewards with a discount. Um, discount is a very important concept in reinforcement learning. It's also a very natural concept to think about. Can you think of what, what the discount would represent in for humans? Do you have an example of what it could be? Yeah.
Yeah. The value of money and time. Exactly. Or the energy that a robot might have. Things like that. Yeah. You would rather get a dollar now than a dollar in 10 years. Knowing that there is some inflation, for example. That's the example of a discount in reinforcement learning is the same. You know, let's say you have a strategy that takes so much time you need to discount it because you're wrong. Robot might lose energy as it's going through it. For example, discounts can vary, you know, but they stay between 0 and 1.
Um, so what is the best strategy to follow? If gamma, the discount is equal to one, meaning, you know, time doesn't matter here if it's longer or shorter. Just want to maximize the return.
Best strategy to follow.
I want you to try someone who hasn't spoken yet.
Yes.
Bounce around. But remember the rule of three minutes. You can't bounce around because you, you will not get to the terminal state before the time allotted is done. But that would be a good idea if this rule was not true.
What else could you.
The idea. It's an easy one. No. Not too hard.
Best strategy for Gamma equals one. And give me also the maximum reward you would get.
People are sleepy today. Yeah.
Go to the recycle. So, Right, right, right. Yeah, that's right. Thank you.
Right, right, right. And then what's your. Sorry, what's your, what's your total reward?
Yeah, that's right. 11. So that's where we get terminal state and we grab our reward of 11. Very good. Now assuming 0.9 for Gamma, we're going to complexify things a little bit. I'm going to walk you through a very simple algorithm that, you know, allows us to sort of determine the best strategy. And we will put our numbers in a matrix. So for instance, we'll define a Q table and Q stands. You know, it's is a, it's a value function where the, the, the name Q learning, Q star. You might have heard all of these things come from Q learning. And so let's say we have a Q table which has the size of number of states times number of actions. So five rows, two columns. In our case, every entry of the Q table is essentially representing how good it is to take action A in state B.
Do you agree that if we had a table with these numbers, essentially we solve the problem? Meaning at any point the agent can just look in the table. I am in state three. Let's look at column one. That would tell me the value of action one. And let's look at column two. It would tell me the value of action two. So I have everything I need to make my decisions.
So that table is really the, the thing you want to find in this exercise. Now the way we will find the table is sort of using backtracking algorithm where we might actually codify the environment as a tree and traverse the tree. So here's what it looks like. I start in S2 and I have two options ahead of me. I can go to the left where I will get a reward of two. It's an immediate reward. The immediate reward is not discounted. It's an immediate reward. Remember the, the formula for R, the immediate reward R0 is not discounted. That would take me to S1. It's a terminal state, so there's nothing to do. After.
Second option, I go to the right and I get a reward of zero. That's my immediate reward. And I end up in State 3. State 3 is not a terminal state. So I can go and do the same exercise from state three. In state three, I have two options. I can go to the left where I would see a reward of 0 and I will end up in S2 or I will go to the right and I will get an immediate reward of plus one. It's an immediate reward. We're not discounting it. I will end up in S4. And from S4 again, I have two options. Back to the left to S3 with zero reward or to the right with the amazing reward of plus 10 and the terminal state of S5. So that's my map of immediate rewards. That's not my discounted return. So what we're gonna do now is we're gonna backtrack up the tree in order to compute the discounted returns. Actually, If I'm in S3 right here.
I see that I can get an immediate reward in S4 of/1. And I wanna compute my maximum return that I can get from when I'm in S3, my maximum return is that in S4 I could get a plus 10, right? But I need to discount that. My discount is 0.9. So I multiply 10 by 0.9. What it tells me is that from S4 I can expect 9 +1, which I get as an immediate reward. From moving from S3 to S4, I can update this number to 10. Meaning from S3, the best you can hope for is a discounted return of 10, which is 1 plus 0.9 times 10.
Everyone follows. Now, let's do the same exercise one step before. In S2, you know, in S2, I have an immediate reward of zero for going to S3 or an immediate reward of two for going to S1. Um, S1 is not going to be worth it. We already know that. Because when I'm in S3, I can actually expect 10, which I have to discount. 0.9 times 10 gives me 9 plus 0 immediate reward from S2 to S3. That tells me that the discounted return from state 2, which is our initial state, is 9.
Go. Follow. Just a simple backtracking. Now I can copy back these so S3. I know that when I'm in S3, you know, I can expect zero immediate reward to.
Sorry. If I. If I'm. If I'm in S2, I can expect zero immediate reward plus a discount times the plus nine that I could expect in S3. And so that gives me values that should cover everything that we have in this Q table. So I do that backtracking. I copy paste all of that into my Q table all the way up here, and this is what I get.
We essentially finish the game at this point. We can look at a Certain row. So let's say I'm in state number three. I look on the third row of that Q table and I see that I have two options. If I go back to S2, ultimately my discounted return will be 8.1. Right. If I actually go to S4 on the right, I will get 10 because I will get one plus 0.9 times 10, which is 10.
So this is a toy example, but it tells you that if you were able to backtrack through the entire environment, you will be able to build a massive Q table and you will be able to give it to your agent to make its decisions.
Yeah.
Sorry, can you repeat?
Yeah, here I'm simplifying. I'm not considering the time remaining. But in practice, you. If, if I remove the time component, so I remove the fact that there's a three minutes deadline before the garbage collector comes, then this would be slightly more difficult because you would have to do a time series essentially of adding the discount times the reward that you collect. Yeah, but I'm simplifying here and that's why I use the three minute rule.
Any question on the Q table?
Super. Okay, so this was the Q table and in fact we can put together our strategy for Yama equals 0.9. The best strategy is still the same. You go to the right and you can expect a return of nine.
Now, one of the most important concepts in reinforcement learning is this equation on the board called the Bellman optimality equation. Oftentimes you'll see it noted as Q star of state S and action A.
Equals r plus gamma times the max of that same function applied to S prime A prime. Let me explain this equation for you because it's super important. This equation is called the optimality equation because your optimal Q table will follow this equation if you have finished the game. This equation can be applied to any state action pair, and it will still be true.
The intuition behind why the Bellman equation is the optimality equation is that if you're in A, if you have the perfect Q function Q table.
And you're in a certain state and you perform a certain action A, you will observe a reward and this reward will, you know, you, you have taken an action. So you would be in a new state and from that new state you could repeat what you just did, right? And because you've done the backtracking and stuff like that, you will get this equation to be true because it's the reward plus discount times the best next action that you could be taking.
Does that make sense? Any question on that? That's exactly the backtracking that we did, by the way. Immediate reward plus discount times the best possible action that you can take in the next state S prime.
The last concept I cover in terms of vocabulary is the policy. The policy is the function that given your state, is going to tell you what to do. And in Q learning, the way this policy is defined is argmax of Q star across the action. So essentially what it says is like look in the table and look at a certain state S. You want the policy, which is what you should do. It's the function that tells you our best strategy. You just look at the two possible actions, which one has the highest Q value and select that action. That's it.
This is a very simple example, but it, it's the core of Q learning that you know, later on you will use policies widely. There's a lot of reinforcement learning algorithms, but this concept of understanding the policy is the function telling us our best strategy in Q learning. It's the argmax of the best Q value in a the given state. It tells you which action to take. That's the core thing you need to understand.
So remember this Bellman equation because we're gonna reuse it.
In a bit. The main issue with this approach of a Q table is that state and action spaces can be super large. And having a matrix that you discover through backtracking and where every time you wanna do an action, you have look up the given states, the possible action, it becomes impossible. Like imagine you using this algorithm for the game of Go, where there's so many states, there's so many possible actions, you can put your stone anywhere on the board, you can imagine how big this matrix becomes and how impossible it is to use. So that's our problem and that's the moment where deep learning comes into play.
So let's look at it.
The, the, the. Oh, actually before I go there, I'm just going to cover some vocabulary. We said the environment, the agent, the state, the action, the reward, the total return and the discount factor. We learned all of that. We saw that the Q table is the matrix of entries representing how good is it to take action A in state S. And the policy is the function that tells us what's the best strategy to adopt. And the Bellman equation is satisfied by the optimal Q table.
So let's get to deeper Q learning, which is what I was about to say is we are gonna frame the problem slightly differently. So instead of using a Q table, we're gonna Use the fact that neural networks are universal function approximators. And we're gonna define a Q function that's essentially a neural network, so that the function can take a state S and an action A and tell you how good that action is in state s. So instead of a lookup in a matrix, you just run a forward pass in a neural network and it gives you the answer. That feels like a better solution for games where there's a lot of states and a lot of actions.
So here is a same problem statement. In the past, we looked for a Q table, and this time we will look for a neural network. One of the things we're gonna do is to define the output layer to have two outputs. So given a certain state as input, think about it as a one hot vector encoding the states. So this one is the example of state 201000. If you pass state 2 in this Q function with multiple layers, it will give you two outputs. One output that corresponds to Q of S action right and the other one Q of S action left. Because it's the two actions. If we had more actions to take, we would just increase the output layer and we might have many more neurons in the output layer.
So the big question is, how the hell are we going to train that network? Because we're not in classic supervised learning, we don't have labels.
So this one is a hard question, but what do you, what would you do given we we don't have traditional X and Y pairs? How are you going to train this neural network?
Because remember at the beginning, this neural network will give you garbage. It will take a state S and it might tell you go to the left or to the right, but you it's completely random. So how are you going to tune it to the level where it makes really good decisions?
Yes.
Assume based on some prior knowledge. Tell me more.
So what are the things we know about this problem right now? What are the rules of the game that we could use in order to.
I'm seeing what you say. You say we could estimate what good looks like, but based on what?
Okay, so reward structure, you're saying that's one thing we have in every game. We have a reward structure for every state that definitely should be used in order to estimate the good. What a good decision looks like. Yeah. The problem is not in every state you will see a reward. And if you look at many games of like Go, you might not see a reward until 50 moves. So what do you do? In this case?
Yes, we run through a bunch of.
Actions and stage and see what the output is and get more data to train the enrollment.
Yeah, so you could. You're actually bringing up a sort of a tree search. Right. You go down the tree, you do every possible action and then you backtrack.
Not every possible action.
So which actions?
Okay, that's, that's. We're getting there. So first possibility is we just go down the tree. In the game of go, you could put your stone everywhere. So the tree already start by a 13 by 13 options and then it's exponentially grows impossible. It's intractable. But what you said is what if there are certain actions that are more likely than others? Do we need actually to explore the entire tree? What's this like? What are you using when you're saying that? How do you determine what action might be better than another one?
Expected return. We're getting close. Yeah, but you know, how do you know the expected return without going through the tree once at least.
Okay, you can estimate it using what?
Yeah, maybe.
So that's exactly what we're going to do actually. But we're going to use the Bellman equation because there are two things we know about this problem. We know the reward structure, which you brought up, and. And we also know that the perfect Q function will follow the Bellman equation that we know as well at the end. The Bellman equation should be respected, meaning for every state, if you want to know the Q value of that state given an action, the way you will get that is you will look at the immediate reward plus the discount times the best Q value from the next state across all actions, that equation will be respected. So those are the only information we have. And we're gonna use them drastically to define our labels and sort of mimic a classic supervised learning approach. So here's what we have. We have our neural network. We have Q s to the left and Qs to the right that represent how good it is to go to the left in that state versus the right. And then I pasted the Bellman equation on top right of the screen. We're going to define a loss function. So let's say for the sake of simplicity, because those are scalar values that will use, you know, L2 loss, quadratic loss, that compares a certain label Y to a certain Q value of a state in a certain action. So what we would like is to minimize this loss function, meaning Y. And the Q value for a given action in a given state is as close as possible to each other. And we're gonna leverage the Reward and the Bellman equation. So let's do two things. Right now, we don't have a Y. So in supervised learning, you will have a picture of a cat. There's a cat. The Y is 1 or 0. Here we don't have a Y. So we have to come up with an estimate of a good Y, at least better than random. So let's say at this point in time, when I send a state S in the network, it turns out that Q of going to the left is higher than Q of going to the right. Which means that today, at that moment, the Q function tells me it's better to go to the left than to go to the right. That is random. At the beginning, it's completely random. So what I'm gonna do is I'm gonna use as my target value Y the immediate reward that I observe on the left.
Plus gamma times the best Q value that I can get. So the best action that I could take in the next step based on my current Q value.
That'S very important. So remember, this target is off. It's not a perfect target, but it's better than nothing. Meaning not only it tells us, hey, there is a good reward to the left, we should consider that in saying that that might be a good move because we're seeing an immediate reward. But on top of that, we also know that at the end of training, the Q value should follow the Bellman equation. So why don't we set the target as the Bellman equation, so we add the discounted maximum future reward. When you are in the next state, so you were in state S, you go to the left. Now you're in state S next left, and you look again at your Q values and you select the best one. Then you add that number here. So there is actually two forward path in that process.
Right?
There's one forward path where you send the state S in Q and you look at the two options, left or right, and you're like, okay, I'm going to the left. And then you're like, I'm gonna compare that value to a target Y. But to get that target Y, I need to do another forward path. So I take my action left, I perform it, I get an S prime state S next, and I send that snext into the Q network. I look at the two options I have, I pick the best one and I add it here with the discount.
So fundamentally, what's happening is, is the following is we have a Q network that's random at the beginning. It has never Observed the rewards. We just know that at some point it will get to the Q, it will get to a perfect, you know, policy. It will get to a perfect Q function. But the best we can do right now is to say as a guide to. For our agent, we will look at the immediate reward and we will look at the Bellman equation, which should tell us a better estimate than where we are right now. And we will try to catch up to that estimate. And then we do that again and again. So remember, every time your Q gets better, it gets better for the next state as well. So, you know, the Bellman equation tells you estimated with the second forward path. And you just keep getting better and better as you're observing more rewards.
The next state.
How would it. So describe the loop you right now.
Imagine this going to arrive like the next stage went to right.
Yeah.
For going to a right, you again need the target.
Yeah. You would stop at that point. So what you. Yeah, this is a good question. I'll show you how we fix certain things. But you do only one step. Meaning you have your Q value at this point and it tells you go to the left and you just want to target Y. So what you do is you put left and you look at your next state, you forward, propagate your next state. You look at the two options. You pick the best. You don't go further. You just use that one step. You look one step ahead. Essentially, you don't look multiple steps ahead. You could, but it would be more computationally heavy to do one more step again and so on. So, yeah, yeah, it seems like you're.
Learning the.
Information about the reward is coming from.
So if people know like how fast that curve is to the true.
It will typically be a function of the environment, the state space, how long it will take to converge. But you're perfectly right that.
As the Q function get better, the estimate Y also gets better. So the two things get better together. Right. Because the Y is based on the Q function and if the state space is massive, you might have a very difficult time training this model. There's better approaches that we'll see later. Yep. Correct. There was a question there.
Yeah. So here I'm just saying, let's say when you send state S in Q, the left happens to be higher than right, but the same happens on the other side. Let's say, let's say the left is worse than right. Then what you will do is you will define your target Y as the reward that you observe on the right. Plus from the next State of having gone to the right, what's the best action and what's the Q value for that pair? And then it will give you the target for that scenario.
So one complication with this training is that when you want to differentiate L, so you want to perform a back propagation, you want to take the derivative of L with respect of the parameters of the network. You want Y to be a fixed thing. Right. Because in supervised learning Y is not differentiable. It's just a fixed number 0 or 1 or a certain number. So here we're gonna simplify and we're gonna say this term that is dependent on the Q network. So technically this term has parameters. So if you actually differentiate it, it will give you a value. We'll just hold it.
Fixed. So we say we do use our Q network to perform an estimate of our Y, but we will not differentiate it. We will say it's fixed. Yeah.
Can you explain why we discount this?
Because going back to the reason we discount is like the value of time. You probably want to say if you can win the game in 10 moves, win it in 10 moves rather than 100 moves. Or if you can get $1 today, get $1 today rather than in 10 years. All of that is why we have a discount here. And the discount is a hyper parameter that you would define as well that would influence the strategy of your agent.
We're going to see it after. Actually I'm going to do a concrete example because it's a little complicated. Yeah, yeah.
No, no, that was. It's a good point. It's not that it's just Q of it's a two by two, It's a one by two. So you have left and right. I was just going down the first case. So I put the state left. Yeah, yeah.
What if the rewards are not fixed? I mean in most games we're gonna see right now the rewards are gonna be fixed by the designer of the game, the human that's designing the game. Um, in practice you could have a separate function that actually comes up with the reward. We're gonna see an example later in the lecture where the reward might be different in different scenarios. And there's a function or a sometimes called a critique that determines what's the reward in a certain state.
Okay, this is yet one last question. And then we move because we're going to see a concrete example is going to be clear.
So when we like hold it fix for laptop, is that what differentiates this from iterating through all of the like possible?
Yeah, yeah. So instead of doing the backtracking down the tree and going over everything, we're saying, we're going to limit ourselves to just pick the best action based on our current understanding of the network. You see, like my network is kind of intelligent. Not great. We're in the middle of training. It says that I should go to the left. And then if I look at the next state when I'm in the left, it says I should go to the right. I will trust it because it's the best I have. Best estimate I have, but I will discount that. And then if you keep repeating that, it turns out that not only your estimate gets better, but your model gets trained. And then ultimately both together get to an optimality equation.
So it's a, it's a funky concept. Right. But you get it. We're, we're going to see examples. Okay. So then once you have been able to use the bellman equation to estimate your targets, you perform classic back propagation and you update the parameters of the network and you repeat that process.
Yeah.
Uh, here is. Concretely, if you were to code it in pseudocode, here is what it would look like to train nor el agent using Q learning. We start by initializing our Q network parameters. So initialization, it's random at first. Then we will loop over episode. As a reminder, episodes are one full game from start to terminal state.
Within an episode, we're going to start from the initial state S and we're going to loop over time steps until we reach a terminal state. So within one time step, here is what we will do. We'll forward propagate the state S. In the Q network, we will execute the action A that has the maximum Q value.
We will observe a reward and we will also observe a next state S prime. We will use that S prime to compute our target Y by forward propagating S prime in the Q network and then computing our loss function. And based on that, we will use gradient descent to update the parameters of the network.
Should be simpler looked at like that. Right?
Okay, so this is the vanilla Q learning.
So to summarize again, the one, the main difference is that we don't have a target and we use our own network to estimate the targets. And the rewards are what's gonna help us get better over time.
By the way, it's okay if you don't understand everything. This is an entire class at Stanford, um, you know, an entire quarter of studying that type of stuff. So we're trying to get the basics within an hour and a half. Two hours.
Okay, let's go a little further now together.
And apply that to an actual game. So here's the game. It's called Breakout. We want to destroy all the bricks. Who has played Breakout in the past? A few. Okay, good. So you have a paddle that you control and you are trying to destroy the bricks. If the ball gets past your paddle, you lost. And if the bricks are all destroyed, you won. That's it.
Let's do it together. What's. What is the input of our Q network? What would you use as input? Through memory? Yeah.
Yes.
Entire screen. Okay, let's do that. So I take. I define that as the state S, which is the input to my Q network. What's the output of the Q network?
Yes.
Good question. We'll get there. I'm going to ask you, but do we have to look at the full screen? Answer is no, but we'll see why. What's the output? Yeah, the game score. No, but we're going to talk about the game score in the back.
Yeah, let's. Let's talk about the output first and then we'll talk about the stuff we can get rid of on the inputs. But what's the output? Yeah.
The actions. Yeah, the actions. So, yeah, it will be the Q values associated with the actions in state S. Remember, it's a Q function. So the output is. We need one value for left, one value for right, and one value for idle. You could make this game more complicated and say we have eight actions. We have a little bit to the left, a lot to the left, a lot more to the left. You know, if you had multiple buttons. But let's simplify and say three actions, either you don't move, you move to the left or you move to the right. So these are the outputs. So now let's get to the question of the screen. Do we need the entire screen?
So you were saying something earlier.
Okay, so you say you need the tray and the bricks. I would argue you need more because there's the walls. And I guess that you could, if you're an expert player, you could know where the walls are. But generally you need a little more than that. What, what, what would be. Obviously things we can get rid of. And why would. Would we do that?
Okay, the score at the top.
Who would remove the score at the top.
But half.
Why would you not remove it?
Why would you remove it?
Okay, you want to always win, so the score doesn't matter. It's true. We would remove the score. So you could actually crop the top you could also crop the bottom. I mean, if it passed the paddle, you don't care about the few pixels at the bottom, you could get rid of them. This is not always true. There are games where the score matters. And in fact, you know, I like football, soccer. In soccer, if you're 10 up, you can park the bus. So your strategy is dependent of the score that you have. Like you wouldn't park the bus if you're losing 10 parking the bus, meaning you ask every player to come back and defend. If you're losing, you would actually do the opposite. You will go all out, attack. So in certain games you want the scores, in others you don't want. And so it's part of the designer, the AI engineer that's working on that to determine what information we need and what we don't need. What else could we do to reduce the dimensionality of the problem and make our computation faster?
Yeah, remove the RGB channels. So do grayscale. Essentially. That's true here. You actually don't need the colors. It's just nice as a user, for user experience purposes. You don't need. I don't think there's different points based on the bricks that you destroy. It's all the same.
Actually, funny enough, this algorithm was used by DeepMind to play a lot of Atari games. And they did a single pre processing where they removed the channels because they said it doesn't matter. Turns out in one of the games, I think it was seaQuest, I forgot which one. The fish disappeared when you did that. And so that game didn't work. The agent couldn't crack it because they thought that the same pre processing could apply to every game. But actually they had to make a slight change it.
Correct. So just to recap, you could do it even better by using a low dimensional representation of this game that describes the game. It's true, but because we want to use a single algorithm for 50 plus Atari games will say, the human sees the screen. We'll just give the screen and it will probably scale better, essentially. But you're perfectly right if you are working on only that game. Okay, so let's do that. We'll. We'll do pre processing. There's one last thing that nobody mentioned, which is history. Because in fact, if you get only one screen, you don't know where the ball is going. So actually you can't solve the game. And the way you fix that is by giving a history of multiple screens, for example, four screens, so that you see the direction that the Ball is going in. So our pre processing function is, you know, called phi of S, let's say. And phi of S is a mix of, you know, you might do convert to grayscale, reduce the dimension, the height and width and also add the history of four frames. And that should be enough. Turns out in most games you will need the history, a little bit of history to know where the ball is going.
Or in this example.
Velocity vector of the ball.
Yeah, you could replace. Exactly. You could replace the history. So multiple screen by just adding the gradient or the velocity of where the ball is going. That's true. But would it scale to every game? You know, turns out this because we know humans look at the Atari machine and they look at pixels. This would be more likely to scale to every game.
Like think about a game where actually seaQuest is a good game, or Space Invaders where you have multiple enemies coming at you. Then you would need to change your pre processing to take into account the velocity of all these enemies. So it wouldn't work the same way. While if you actually give the pixels, you actually from the pixels get the velocity of all your enemies and the directions they're going. Okay, so this is our pre processing. I'm gonna refer to it as FiOS. And our deep Q network architecture, because we're working with pixels, is going to be a convolutional neural network. Don't worry if you haven't learned it it yet in the class, but it's a bunch of conv and relu activations. Um, and then we end with a fully connected layer that gives us the 3Q values for the different actions.
So nothing special here. Now we're gonna go back to our vanilla training. So this one that we saw together earlier and we're gonna look at tips to train reinforcement learning algorithms. Those tips are not specific to Q learning. Some of them are applied to a lot more than Q learning and they're very important to know. Um, and they're part of the reason reinforcement learning has worked better in the last few years. Um, so one of the things that's pretty simple that we forgot to do, um, is the pre processing that we just did. So anywhere I had an S, I'm gonna instead run S Through the pre processing step, I'm gonna use use phi of s. So I initialize instead of S with phi of S, I start from the initial state phi of S and then I forward propagate phi of S. I, I get the Q of that pre processed state in action A and et cetera, et Cetera. And then when I get my next state. So let's say I look at my current pre process state by forward propagated ones. I see the 3Q values. In our case, the 2Q values, 1 of them was better than the other action writes. Then I get my next state, S prime. I wanna pre process that state as well.
So that's pretty straightforward. You just replace all of that.
The second thing we forgot to do is to keep track of the terminal states in our pseudocode. There is no concept of terminal states. It's pretty easy to add. You would probably just do an if else statement. You would create a boolean to detect terminal states. So let's say your boolean is terminal equals false. And then as you loop over the time step of a single episode, every time you're gonna check is the state that I'm going in based on the action I'm taking a terminal state. If it's a terminal state, then get out of the loop. You know, there's nothing else after. The one thing that you need to be careful of is if it's a terminal state, then your target is not. The bellman equation is just the ter, the immediate reward. Remember, you get to the terminal state, you get a reward of 10. There is no Bellman equation to apply. It's just 10. It's immediate reward, there's no discount, et cetera.
Okay, so these are fairly easy changes. Now we're gonna look at a new method that will enable more data efficiency. It's called experience replay. One of the a couple of issues with the way we've been training so far is one, the correlation of successive screens. It's like imagine in the Atari game. You have the ball that's in the top left corner and it's traveling to the bottom right of the screen. You have like many, many time steps that are essentially the same. It's all the ball traveling in the same, in the same place. So you're actually training repetitively on something that is not that meaningful. You don't need to just train on a batch. The equivalent in supervised learning is, let's say you're trying to differentiate cats and dogs. And you train on a mini batch of cats, then you train on a mini batch of dogs, then you train on a mini batch of cats. And you will never converge. It will just index too much on cats and then index too much on dogs. So you want to add some sort of a.
Experience replay concept that we see in order to create more mixes in the data and get more diversity the other thing that is important is in our current training process, we are not reusing our data. Like you experience something, you immediately train on it, you never see it again unless you re experience the same thing sometimes in the future, which might or might not happen. Experience replay is going to help us to keep experiences in memory and maybe retrain on them on a regular basis. So that one experience might be useful multiple times. Which intuitively makes sense. Like maybe you do an experience, you get an amazing reward and you don't wanna forget it, you wanna retrain the model on it on a regular basis. It's more data efficiency. So here's what it looks like. Um, the current way we were training was we're in a state, I'm just gonna state state instead of preprocess state. But it's pre processed. We're in a state S, we perform action A, we get a reward R and we get into the next state. From that next state we perform another action A prime, we get a reward R prime and we get into S second and so on, you know, and so on. And each of these would be called one experience. It's one iteration of gradient descent, it's one experience. So right now we're training on these experiences. So the training looks like I train on E1, I update my parameters, then I train on E2, I update my parameters, then I train on E3, I update my parameters. Those are highly correlated because they're part of the same episode. And as I was saying, with the ball traveling in one direction, that might actually not be that helpful to train on all of these, you know. So instead what we'll do is we'll use experience replay where we will collect our first experience. But instead of training on it, we will put it in a memory called the replay memory D, we'll put it in there and then at every step we will sample from that memory to decide what to train on. So of course at the beginning if we just have one experience in the memory, we will train on that experience. But over time you will see that we get more diversity and reuse out of our experiences. So for example, let's say I experience E2, I put it in the memory and then instead of training on E2, I'm gonna randomly sample from the memory. I might get E1 or I might get E2, then I experience E3 and I put it in the memory and I might get one of the three. You know, this is the vanilla experience replay. In practice there is more methods like prioritized sweeping which might tell you which experience you want to weigh. Maybe some experiences had a higher gradient so you want to prioritize them more often, things like that.
So all in all, this is what our training looks like with experience replay. We experience E1, we train on E1, then we the next training iteration is not on E2, it's on a sample from E1 and E2. Either or the third experience is then put in the replay memory but we don't train on it. We train on a sample from whatever is in the replay memory and we repeat. And that is more sample more efficient, allows more reusability and less cross correlation in our training batch.
So that's called replay memory and you can use it with mini batch gradient descent. Note that you still experience in the direction that the game is played. Like we still go and take the action as expected. We just don't, we don't necessarily update our model parameter based on the action that we ended up taking. We put it in the replay memory. We may train on it later.
Okay.
So here is how it modifies our vanilla setup. We've added an experience from state S to state S prime to the replay memory. You know, like let me walk you through it again. Within one time step we forward propagate our state into the Q network. We execute the best action given the Q values. This gives us a reward and the next state, the next state is pre processed and then instead of training on that, instead of training we just add that transition to the replay memory and instead we sample randomly a mini batch of transition from the replay memory and we train on those and we redo the same thing again and again.
Yes, this is reclaimed bias towards the start of the game.
So aren't you more likely to train a start game that you were learning something from chess? Like you would sometimes want to.
Yeah, you would, you would within one episode. But you know, if you play multiple chess game, your replay memory will get already bigger. So then you would see some end game, some middle of the game, some early games. Yeah, good.
And in practice it's actually useful because you might imagine that in a chess game, you know, all of us, let's say if you're a beginner, you, you see a lot of beginning of the games. You actually people that are beginners, they're good at openings but they're bad at endgames because they don't get to play a lot of endgames. Uh, well, that type of approach could be useful. You can retrain on endgames more often and you know, the A More advanced version of the replay memory would also weigh the experience in the replay memory based on how much the gradient is going to be. So if you have an experience that actually was super insightful, you can weigh it higher so that you, you, you prioritize grabbing it and retraining on it, essentially.
So let's say you blunder in chess. You might actually want to re see that blunder later so that you don't do it again.
Okay, so these were all the different methods. Another one that's very intuitive and very important is when during the training process, our agent gets stuck, gets stuck in a local minima. Uh, here is how it would work in practice. Uh, you start in initial state S1, and you have three states ahead of you. If you take action A1, you go to state two, which is a terminal state. If you take action and you get a reward of zero. If you take action A2, you get to S3, also a terminal state, and you get a reward of one. And if you get action A three, get to state four, terminal state with a reward of a thousand. So of course, to us it's obvious that we would want to explore the state number four. It's pretty obvious in practice. Let's say you update you, you initialize your network, and in the first forward path, that's what you get. First forward path, the network is random. You get Q value for action 1.5 for action 2.4 for action 3.3. What does that mean? It means the agent is saying, I'm gonna go to action one. So I take action one and I see an immediate reward of zero. Right? Because it's a terminal state, the Bellman equation thing doesn't happen. I just have the immediate reward which becomes my target Y. And so I perform a gradient descent update to say these Q values should have been zero. So I convert this Q value to zero. Now second try. This time the Q value is saying, take action two. It's the highest Q value. I take action two, I have an immediate reward ahead of me. That's one. Because it's a terminal state, there is no second discounted future reward term. So I just take y equals 1. I perform my gradient descent updates and this converts to one. And then third time, the agent is still saying go to a two. Go to the take action A two reward of one. Good. That's what you predicted. Nothing to do, just keep going. We're done with training. We're stuck. We never visit the state we actually wanted to visit.
Okay, so that, that, that wouldn't work for us. We will never visit that state using our current algorithm. Does that make sense why we wouldn't ever visit that state in practice? This is a big issue. The analogy of this concept of exploration versus exploitation is when every day you take your bike and you cross campus, you have a favorite route and turns out that the more you take that route, the better you get. Every time, like you get a little faster, maybe your turn is faster or something, or you can predict how many people are going to be at that roundabout and you know how to take it in the wide way so you go faster. We've all done that.
That's exploitation. You exploit what you already know and you get better at it. But maybe there's another route that you're not thinking of that's pretty. Instead of going north from campus, you go south. And maybe it might be better. You will never see because you don't have the courage or the patience to do it. That's the difference between exploration and exploitation. In practice, a good model would be able to handle both. To exploit when it. To exploit, to explore when it needs to explore. The way we do it in practice in our pseudocode is to inject some randomness. So for example, when we are looping over time, step with Probability Epsilon, let's say 5% take a random action. So from time to time, on average one time every 20 times you take a random action, it will allow you to visit maybe a new path. The analogy in chess is, you know, you might use a creative move from time to time that might be worse today, but might allow you to learn something and to get better over time.
Yeah.
The last example we just covered, could we resolve that by just setting the initial Q values to infinity?
Setting the. The. Couldn't we resolve this problem by setting the initial values from into infinity? Well, the problem, if you set the initial values to infinity. So you would say instead of randomly initializing your network, you initialize it in a way that the outputs are equal to infinity.
Yeah. So that we wouldn't get the issue of where, like the Q value of action 3 from state 1 is less.
Well, in practice, if the 3Q values are infinity, then you can't make a decision on the spot. So you're saying just pick one randomly because if the three are infinity, you, you can't decide which one to take. Right.
And also if, if it's infinity and the reward is one, I mean, if it's a really large number and the reward is one, your gradient is going to be massive. Right. So it's gonna. I guess the loss function is gonna be massive. And I don't know, I imagine it would be really hard to train it. But in practice, you start with the random initialization, because this might be one example. But you know, if in the game of chess.
Actually the reward is one at the end and zero all the time, or maybe the reward is a thousand at the end. And when you lose your rook, it's. It's a negative reward. You can't predict what the reward structure is gonna be. You want an agent that is able to adapt to it, and it's better to find a method that can scale to different environments, essentially. Um.
Okay, so this was epsilon greedy action, which is adding some randomness with probability epsilon. Take a random action. Okay, so adding all our techniques, because we get good at training reinforcement learning algorithms, this is what we have. We initialize our Q network parameters. We have a random network. We initialize our replay memory D, and then we loop over episodes. We start from an initial state. We create a boolean that allows us to detect terminal states with probability epsilon. We're gonna take a random action, otherwise we're gonna to follow what we know, which is forward. Propagate the state in the Q network. Take the action that has the highest q value. That allows you to observe a reward and the next state. Take that next state forward, propagate it again.
Instead. And then, and then instead of. Oh, no, sorry, sorry. Observe that next state. Add it to the replay memory, sample from the replay memory, and then train on that sample. And in the process, you will need to do another forward path because you need to estimate your target Y using the immediate reward plus the bellman equation plus the discounted future reward.
Okay. Are you experts at Q learning?
Okay, good. Sounds good. And here's where we get. At the end, you can claim proudly you have trained an attire. It's not that complicated, as you can see, other than the bellman equation piece. Turns out the agent has discovered that it can send the ball on the back and it's actually much easier to finish the game like that. Which is quite interesting. You know, a good player would know that you can dig a tunnel and you can finish the game without too much issues. Yeah.
How do you quantify when the game has ended?
Yeah, well, first you would start seeing the model get too good rewards as it plays like it manages to get really good rewards, while earlier it might not. And so that's probably your best guess for how good the model is in practice. If you're alphago you can also test it against the best human in the world and you can observe that they're losing against the model.
I'm saying you have like a bunch of different chess engines, and some of them are way, way better than others. They have different infrastructure. Maybe they're both based on reinforcement learning, and at the end they maximize both of their rewards. Yeah. So how do you know which model is actually doing better?
You can get them to play together.
So you have no idea?
No. You. You could, you could actually monitor the loss function. And look at. Is the Bellman equation respected? If the Bellman equation is respected, then your model is really, really good. And then we, we're going to see an example of competitive self play where you get the model to play against other models. And then over time, as you watch them play for thousands and thousands of time, you can tell which model is ahead of another one. You can then sort of copy paste the best model into the other models and then make them play again for many times. And because you have the epsilon greedy approach, one of the model is naturally gonna get better than the others because of the randomness that you add.
Um, okay, let's look at a few examples and then we'll spend 20 minutes on the RLHF. Um, here are other examples. This is Pong, which is 1v1. SeaQuest.
Which is an underwater game. And then the one that maybe more of, you know, Space Invaders, very popular game as well. So the impressive thing that they showed is that you can actually solve many games with the exact same algorithm, no tweaks, which is quite impressive.
Let's go a little further and talk about advanced topics. Here is a game called Montezuma Revenge. This game is particular because you're controlling a little character right here. And this character is trying to go and grab, let's say, this key right here. And it has some obstacles or some enemies that it needs to take care of. What. What do you think is going to be an issue if we apply what we just learned to this game?
What makes this game especially hard in comparison to, let's say, chess or Go? Yes.
Yeah. The reward is very delayed. Like if you start with a random network, what are the chances that the network is gonna figure out that to get to the key, it actually should go in the opposite direction? It should go in the opposite direction. It should jump down here, it should catch the rope. The rope will probably allow the character to go to the ladder. It goes down the ladder, it has to go jump up this enemy. My guess is it's in an enemy? I'm not sure, but I think it's an enemy because of the color. And yeah, I know that in gaming, if it was green, it might not have been an enemy, but if it's gray or red, it might be an enemy. And then go up the ladder and grab the key. The chance is very low that the agent is going to make that successive good decisions to get there. You're right. Why is it use. Why is it easier for a human to actually solve that game?
Intuition, prior knowledge. So for example, when you look at this game, even if you have never played it, my guess is you would know you can go down the ladder because you know what a ladder is. Or you can see this little rope and you're like, I'm gonna catch the rope, I'm gonna jump and go to the other side. And you look at this little monster and you're like, I better not touch this monster. Or if anything I will jump on top of it. Because you've played Mario, let's say. So all of this is human intuition. Sometimes you would call as a baby survival instinct. Like you throw the baby in the water and suddenly it flips and it can, it can swim. Those are things that are to a certain extent encoded in our DNA, but at the very least encoded in our experience of doing other things that have nothing to do with this game. And so the, the problem here is called imitation learning is is there a better way to start our network than a random initialization that allows the network to, for example, guess that this is a ladder. And turns out that if the network knows that it will be more likely to get to the reward first and then learn from that reward and then get better over time.
The other part that can also use human knowledge, which is what we're gonna see together, is reinforcement learning from human feedback. Where you have an analogy here, which is you can train a language model and it might be completely misaligned with what actually humans care about. How does reinforcement learning help in those situations? That's gonna be the next topic in the last part of the lecture. Okay, let me show you a few other results quickly. Today we talked about DQ and deep Q learning. In practice there is a lot more reinforcement learning algorithm. But you got the gist of it. You got the concept of making good sequences of decision epsilon, greedy exploration, exploitation, terminal state, starting state, all of that. You, you got the one, the one algorithm that is very popular right now is called ppo Proximal policy optimization. There is one that is even more Popular right now. That's actually from a year ago at Stanford called DPO that we won't study in the class. One of the things to know about ppo, just, just to go over it really quickly and, and I, I pasted two important papers from Schulman et al a few years back. Trust TRPO and ppo, um, is that it is not a value based algorithm. So in Q learning, you learn the Q values and then you define your policy as the arg max of the Q values. In ppo, you learn the policy directly, which is a more probabilistic method. Um, it also works well with continuous spaces. If you look at the Q learning, we learned one output for one action. If you actually have a game that has continuous action, like autonomous driving, where it's not like just turn the wheel to the right or to the left, it's like what degree you turn it, it's continuous, then Q, the QN would not work well or you would have to granularize the number of action a little bit to the right, a little bit more, a little more, which would not be really useful. Instead you would use ppo. Yeah.
Yeah. So question is, how do you define the reward in the qn? Different reward structure will lead to different types of, you know, agent strategies. But you're right, for the game of Go, you could actually define the reward as one if you win and zero if you, you don't win. That's it, you know, every move will be zero until the last move is a win. In chess, you might actually do intermediate reward because you want to tell the, you want to tell the agent that it's good to kill the opponent's pieces to get rid of them. You could also do end to end and say, I don't give any intermediate reward, I just give a final reward, which might be more complicated to train on, but it might actually lead to a more optimal strategy because in fact you could actually win without taking any piece from your opponent. So other things about PPO is, you know, it's more probabilistic. It has a concept of an expected advantage which at every steps, instead of telling you how good that action is, it would tell you how much better it is than random than, than the current state. Like how much better would it be to do certain things versus what you would have done otherwise. I'm not gonna go into the details, it's all in the paper. But those are things that, that are important. Here's a few examples of PPO. So this example on the left is from OpenAI a few years back where you can see it's a continuous space where the agent is being bullied a little bit, but it's trying to grab the rewards. But it's also subject to external forces that are sort of throwing balls at it. It's a little bit mean, but.
Um, you can imagine that this is a continuous space, meaning you're controlling the nodes, you're controlling the joints of the agents and you're controlling the forces, the angles. And so it's a slight. That's why PPO would be better in that case.
Super. Uh, here's a competitive self play which I really like.
Where you have agent play with each other and this is the sumo game. Push the opponent outside the ring and you get a reward.
So actually it's interesting because you're seeing some emergent behavior which is they attack each other's feet or they lower their center of gravity to be more stable, for example. Yeah.
Yeah, yeah. It's versions, sometimes different initializations, for example.
So. No, but. Good question. So oftentimes what OpenAI would do back in that time is, is they would create copies of the same model. They would initialize them differently and they will let them learn and turns out one of the model will get better than the others and then they will copy again that model to the rest and do the same thing again and again. Pretty much.
It's kind of funny, isn't it? That's a good catch.
That's a good goal.
Could watch that for hours. Okay.
They'Re a little awkward, you have to say, but. But it works. Okay, so this, you know, I let you watch the video, it's going to be shared. But here's another set of games that are even more complicated that I mentioned early on. OpenAI 5, which you can think of an equivalent of League of Legend Dota where you have 5v5 game so you have to collaborate, etc. Which makes it adds like literally one additional degree of complexity. And Starcraft Alpha Star from DeepMind is an example of where the observation is not the entire states, you have fog. And so that adds another layer of complexity. Not going to see that together today.
I would encourage you to look at the AlphaGo documentary on Netflix if you haven't. Who has seen it already? Nobody. Okay, well you can now watch it with a different eye. Understanding, reinforcement, learning. And at some point in the, in the documentary you will see that AlphaGo makes a very odd move, a very creative move and people are like, I don't understand that move. Even the top Researchers or the best players would say in the video they don't understand that move. It turns out that that move is very unintuitive for humans because as humans, we are trained to maximize our chances of winning. Like, literally, if I can eat all your pieces in chess, I will eat all your pieces. And if I can surround your stones in go as much as I can, I will do it. The agent is just programmed to win. So that move actually look counterintuitive because the agent doesn't care about winning by one or winning by, you know, 20 stones. It just cares about winning. And that move specifically put the agent in a good place to win by a small margin. Yeah. So that's an example of an insight that you will learn you, you understand from this class and you will see in the, in the documentary.
Okay, I think we have 10 minutes. I'm, I'm just gonna introduce reinforcement learning from human feedback because it's a more.
Modern topic that is very trendy right now. It's important to know. And so let's look at it together. We're gonna start by recapping how language models are trained in a nutshell. And then we'll see what self, what supervised fine tuning looks like. We'll talk about how do we train a critique model, a reward model, and then finally, what RLHF looks like and why is it so trending in the news? So.
Our training objective for language models is next token prediction. Right. We've already talked about it in a former lecture. The idea is that I will get some inputs. I'm reading Wikipedia, let's say, or some sort of a text online, and I read a sentence and I predict the last token, and I do that again and again. So, for example, deep learning. And then deep learning is. Deep learning is so. Deep learning is so cool. And that's it. So you get the idea, right? You always predict the next token, and then over time, it forces the model to explicit emerging behaviors, and it understands the connections between those concepts. Concepts. And it's really good at generating texts. We compute a loss function. You're actually gonna study this loss function in C5, so I'm not gonna talk about it right now, but you perform, you know, a gradient descent loop, and this is how you get your first pre trained language model. Get a pre trained language model. You can call it on a text or a prompt, and it will continually generate, and you call it again and again and again, and it generates, generation, generate, generates. Everybody's comfortable with that, right? Okay, so that's how we trained a language model. But there is a couple of problems. The first problem is that online data does not reflect helpfulness.
So to give you a concrete example, what you might find in the training set is something like deep learning is so cool. When actually what you might find in practice is people asking what is deep learning?
So the data is not really reflective of. You want an agent to be helpful and that's a problem because the model was trained to continue text rather than answer questions. In practice you would see it's a big problem. Another problem is the model has no concept of good, polite or helpful yet. And to give you a concrete example, you might actually ask a pre trained language model My laptop won't turn on, what should I do? And then the model responds because it has read it on Reddit or on Wikipedia is laptops sometimes don't turn on because of power issues. Which is not what you ask. You ask what should I do? And in fact a better answer would have been check your charger if it is properly connected or the outlet works. If that's fine, try holding the power button for 10 seconds. If it still doesn't start, the battery or motherboard may blah blah blah blah. That's a better answer. That's what you want a language model to do nowadays. Um, and the model um, can give you factual text because that's what it's been trained on but it's, it doesn't understand being helpful or um, having an answer that looks like a human like answer.
So our solution to it will start with using supervised fine tuning which is going to be learning from human written demonstrations of helpful behavior. And then we get to even further and use RLA hf which will optimize not only for human written sentences or paragraphs, but for preferences and the word preference is the keyword. Let's talk about how we can improve our pre trained model with supervised fine tuning. I take that we want to align models with human written responses.
And the step one that we're gonna lose is to build a data set. Let's build a data set of human prompt response pairs. So what actually OpenAI is going to do, I'll explain it in a second, is it might collect some of the prompts that we all use and then ask humans to respond to those prompts and put that in a data set. It might also ask separately experts to write really good prompts and then answer those prompts. It's a fully human made dataset. And then we use that dataset to fine tune our pre trained model and by now you've learned fine tuning in the online video. So you know what I'm talking about using supervised learning. So what it looks like is I take my pre trained model that I just told you how we train and then I give it a prompt. Explain deep learning to a beginner and I also will concatenate to it a response, a good response written by a human. Deep learning is a type of machine learning that uses neural and then I expect the model to come up with the word networks. So it's literally do whatever we did to train the pre trained model, but we do it on human written prompt response pairs.
And if you do that many times and you use the, you know, the same loss function, how far the model's response is from a human response, uh, you do that many times and you will get SFT supervised fine tuning. Um, but it has some shortcomings.
One of the shortcomings is it is data that is extremely costly to collect. In fact, I believe in the first version of that instruct GPT there was only 13,000 prompt response pairs. And turns out it did really well. Despite that.
The second aspect is it's unlikely to generalize well because you're again, you're not doing reinforcement learning here, you're doing supervised learning. And so you're just showing a set of examples, 13,000 examples that you want to learn. But it's what tells you that it will generalize to an unseen prompt that will come up from your user base. And so this approach SFT really teaches the model to imitate good behavior from humans. And that's the key. It's, it's imitation. It is not preference optimization.
To do preference optimization, that's where we're going to train a reward model and we're going to do proper RLHF. So let me talk to you about the RM reward model and then I'll tell you about RLHF. In a nutshell, the problem of RLHF is to align not with human responses, but with human preferences. So what ha. What, what's gonna happen is we're gonna train a separate model to predict which responses human prefer and we're gonna call that model the reward model. It's a separate model from whatever we've trained before. The model is gonna use data from labelers, so you're Gonna show labelers 2 or more responses to the same prompt and those responses will be sampled from the sft. So your best model right now is esfd. You will sample three or four responses and you know how we sample, right? You, you can tweak the temperature, you can select not only the top priority word, the top, the SoftMax layers number one word, but you can sometimes sample differently and you will get a variety of answers and then you will ask a human labeler to say answer B is better than answer C and answer C is better than answer A and answer A is sort of equal to answer D. Let's say, um.
They will be asked which response they prefer. And it can get more complicated. It doesn't have to be just a simple ranking. You, you have multiple liker scale methods and so on. Um, but the, the point is that you will collect those pairwise comparison that we will call preference data and you will use it to train a reward model which is initialize from your sft. So your SFT is here. It's your best model to date. And you're gonna modify the last layer. So the softmax layer at the end of a language model that will tell you this is the token we should output or this is the word we should write. Um, instead of that, you'll get rid of that layer. You'll put a scalar value as output. You'll put a linear layer with a scalar value that will represent the reward head. It will predict the, the reward, you know, which is a proxy for the preference of the human. The way you'll train that reward model is you'll give it a batch of two, um, you know, you give it a prompt X with a response A and the preference of the user and you'll give it the same prompt with a response B from the SFT with the preference of the user. So here the user is saying response A is better than response B. And so if you actually were sending that in this model you will get a predicted reward for the preferred answer and a predicted reward for the least preferred answer.
That allows you to train using.
Loss function that I'm not going to cover. Given our time sensitivity, the loss function will encourage the model to assign higher rewards, two preferred responses. So you're trying to dissociate the higher reward better preference from the lower reward for lower preference. And it turns out that if you do that many times, you will have a reward model that given a prompt and a response will be approximating human preference. So you've just trained a critique that represents your humans. It's a proxy for what humans prefer. It's been trained on a lot of human preferences. The reason it's better to use a model than actual humans is because we can use it widely on all sorts of inputs and it can scale from a data standpoint. Also note that this method is better than SFT because it's way easier to ask humans what's your preference between those two things? And to ask them to come up with answers to prompts takes way less time. And if you've used ChatGPT, you've probably been asked before to tell them which response you prefer.
So once trained, the reward model replaces the human as the evaluator during reinforcement learning with from human feedback. And reinforcement learning from human feedback is very comfortable for you. Now I will, I will show you what it looks like given the Q learning algorithm we learned. But essentially we, we have first taught the model what good behavior looks like with sft and then we built a reward model that can tell us how good an answer is according to human preferences. And the RLHF approach is where we will let this model practice get scored by the reward model or the critique and update itself to produce higher scoring answers so more preferred answers. And it's the same as the games we've seen together, but some things differ. So I just pasted here the exact setup that we've learned together for reinforcement learning. The differences are the following. You know, our objective is still to maximize expected reward that is produced by the reward model aligned with human preferences.
The agent is the language model being fine tuned.
The environment is the space of possible prompts and continuations. It's any, any text that you can encounter. The state is the specific prompt plus the tokens that were generated so far. The next state is one more token added and the action is the next token that is chosen by the agent or the model, which is of course determined by the policy.
And then the reward is estimated by the reward model that we trained to represent human preferences.
In this case, one episode is one full prompt. So imagine that you get a prompt and you start generating and you go through this reinforcement learning loop and you observe the rewards and then you try to maximize the future rewards. And then at the end of training you end up with having your pre trained model turn into an SFT and your SFT turn into a way better model using RLHF.
Okay, so a few things to note to end on this. The model does not get a reward at every single token. It gets a reward at the end of a sequence when the completion is finished. Because the reward model was was asked to rate prompts and responses together. So you need to finish the generation in order to see what's the reward. And so, again, going back to making good sequences of decision. That's exactly it. You want the model to make enough good sequences of decision so that the response is preferred by the critique, which represents a proxy to the human preferences.
So all intermediary rewards are typically zero, and that makes it a very sparse reward. Episodic tasks, just like a game of chess, where you only get a reward when you finish, assuming you're not defining intermediary reward. So you only know if you did well at the end, and you have to then use that information to update your network and get a better proxy for it. Super. There's a very nice video. We're not gonna play it for the sake of time, but I will send it online. It's from four days ago. It's, you know.
Former Stanford students Andres Karpathy, who is very thoughtful and articulate and was explaining four days ago why reinforcement learning can be terrible at times and that human minds work way more efficiently. And so I would encourage you to watch this four minute video because he's very clearly outlining why reinforcement learning is still not great, even if it's the best thing we can use in many ways.
More from Kian
Library of Minds: 100x AI Engineers Will Be Treated Like Elite Athletes | Kian Katanforoosh (Workera, Stanford)
Stanford AI Expert: 71% of People Won't Survive the AI Shift — Here's the 30-Minute Fix
Stanford CS230 | Autumn 2025 | Lecture 4: Adversarial Robustness and Generative Models
Stanford CS230 | Autumn 2025 | Lecture 2: Supervised, Self-Supervised, & Weakly Supervised Learning
