Mastering RLHF with AWS: A Hands-on Workshop on Reinforcement Learning from Human Feedback

Hi everyone, my name is Diana Chan Morgan and I'm part of the DeepLearning AI team bringing you all together for all things AI community and events. Today we are very lucky to have a hands on workshop with some special speakers from Amazon Web Services talking about mastering RLHF, also known as Reinforcement Learning Human Feedback. While we get started, we would love to hear from you on the live chat where all of you are dialing in from and what courses you've taken so far at Deep Learning. This event is designed to equip you with all the skills and knowledge required to excel in reinforcement learning applications and effectively leverage human input to enhance AI systems. Whether you're a seasoned data scientist, an aspiring AI enthusiast or a business professional seeking to understand the potential of reinforcement learning, this workshop offers a unique opportunity to unlock its true potential. We will also be dropping a link in the chat for you to ask questions for the speakers and to vote on all other participants questions. This workshop is also based off the foundational learnings of Deep Learning's course on generative AI with LLMs built in collaboration with the Amazon Web Services team. Everything covered in the workshop is presented as continued education from the course and we will also be dropping the link in the chat now to introduce our speakers and fortunately our course instructors as well. I'd love to introduce our first speaker, Anjia Bart.

Hi everyone. Hi.

Anjia is a principal developer advocate for AI and machine learning at AWS. She is also co author of the O'Reilly book Data Science on AWS. You may have seen her speak at AI and ML conferences, events and meetups across the globe. Our second speaker is Chris Fragley. Hey Chris. Chris is a principal Solutions Architect for AI and Machine Learning at AWS. He's also the co author of the O'Reilly book Data Science on AWS. Chris is also the founder of the global Meetup series titled Data Science on aws. Without further ado, Ante and Chris, why don't we take it away?

Absolutely. Thanks so much Diana. Thanks for having us. And thanks everyone who joined us today to learn more about RLHF, which stands for Reinforcement Learning from Human Feedback. All right, quick agenda for today. This is a hands on workshop and Chris will walk you through the actual code, how to run RLHF, how to prepare the data, how to use reward model and how it all comes together to give everyone the foundation. I'll gonna walk you through a couple of slides in the beginning to give the theory and answer a couple of the questions that you submitted before so we already had a peek at those, so I'm trying to get to as many of those while I'm presenting. And then once I'm done with the first part, I'll hand it over to Chris.

Cool.

All right. Yeah. Chris, everything ready from your side? You have an exciting hands on card.

It will be ready. Yeah. Yes, we're hanging on it right now.

All right, let's switch to my. Yeah, let's switch to my slides. All right, let's get started. So as Diana mentioned, this topic and a couple of the slides I'm showing here are part of a course we just launched together with Deep Learning AI on Kousa Sarah called Generative AI with large language models. Here you can see the instructors and also Dr. Andrew. We're super happy to work with him on this course. And also big shout out here to our colleagues Shelby and Mike who also take part as instructors and help develop this course. Here's a quick link to the Deep Learning AI site and the course if you want to have a look at it or maybe even enroll after today. I think we're around 70,000 enrollments already. Super happy and thanks everyone who might already have peeked into the course. And for those of you who haven't yet, let's have a quick look today into one of the topics we're covering. That's RLHF. Also here's a bitly link glm if you don't want to type the full link, but I think we're also posting it here in the chat window. All right, let's jump into today's topic. So in the course we're not only covering reinforcement learning, we're really kind of looking at Generative AI from an end to end project lifecycle. And you can see here we're starting with defining the problem, talking about the use cases, how you can choose the right model. And then here in the middle we're diving into the how to adapt and align your model project phase. And this includes a couple of different things, right? From just using prompt engineering to kind of guide the model towards the right outputs. And then a big section of the course will focus on the fine tuning. So how can you customize the model with your data for your specific use case? And fine tuning, as I said, is about customizing the model, also helping it to understand human language, the prompts and understanding tasks better, also responding in a little bit more natural sounding language potentially. But there's a problem with that, right? Human language as we know sometimes has different tone, sometimes the models pick up language if it's trained on data from the Internet that we don't want to have it replicate to our users. And there's a couple of examples, not just the tone and models behaving badly, but also, for example, one of the non official evaluation criteria, just a personal one that I do. If I'm working with a new model, I'll try to tell me a knock knock joke. So I'm German background info. I moved to the US and I'm still struggling with knock knock jokes. So I'm asking my foundation models to come up with one. Let's say it responds with clap clap. I mean it's kind of funny, right? But it's not really helpful. It's not a real knock knock joke. Also, sometimes the model might respond with something that's completely wrong, right? Can coughing effectively stop a heart attack? Of course not. But maybe the model just assumes it is and really confidently say that. So it's not really honest in this case. And also sometimes the models can provide bad advice or even like how can I hack my neighbor's WI fi? It definitely should not give you tips and best practices. In this case it should just stop and say, hey, I will not answer this. So it can create a little bit of harmful content, sometimes it can be hallucinate, create misleading answers, and sometimes just not helpful. So how can we address this? Right, and those three points, helpful, honest and harmless by the way, are often referred to as the three H's hh. So whenever you hear that that's referring to those overall kind of alignment criteria, you want the model to be helpful, you want it to be honest, and you want it to be harmless. All right, and this is exactly why we're doing RLHF. We're trying to align the model with human feedback to guide it towards harmless content, honest content and helpful content. Let's talk a little bit about reinforcement learning from human feedback and what's happening here. Usually you start with an instruction fine tuned model so the model that already understands how to use instructions and how to respond to different tasks. And then you're applying this RLHF technique as a second fine tuning step to align the model further across those criteria. Again maximizing the helpfulness, the relevance, minimize any harm and avoid the model engaging in any dangerous topics. So before we go into the details of RLHF, I just want to make sure everyone here is familiar with the terminology. If we look at classic reinforcement learning, you might have heard about the terms. We're often speaking of an agent and an environment. The goal here is to maximize the reward. So the agent will take actions in this environment and then based on the action's effectiveness towards a goal, it receives a reward. So it takes an action and the action will put the environment in a specific state. And I'll explain what that means with an example in a bit. And then based on how good that action was and the resulting state towards the end goal, there's a reward. And it's much easier to understand if we use an example. Let's say we want to play Tic Tac toe. The agent would be the player. And the objective here is clearly, hey, I want to win the game. The environment here is our 3x3 board where we play the tic tac toe. And the state is the configuration of the board at any step we're taking. And the action space is the allowed moves I can do. So wherever I can put my X or put my circle here. So how the agent figures out the best way to win this game is by using what we call a policy, a reinforcement learning policy, which is often a model that learns what actions lead towards winning the game efficiently. And based on all this input, like what's the state given the action I've taken and what's the reward, the policy will evolve over many iterations and figure out actually by learning trial and error, which one is the right policy and the right strategy to win the game and a whole play out, like here, like I do the whole iterations until I win or lose. The game is called a playout, or if we're talking about language models, often called a rollout as well. So we're not playing Tic tac toe, unfortunately, we want to fine tune large language models, which is also fun. So how does this concept here apply in the context of LLMs? So we still have an agent and an environment, but in this case here, the reinforcement learning policy is our instruct fine tuned model. And the objective here is not to win a game, but to generate text. And we want to generate the text in an aligned fashion. So what we discussed before, if we want to align for the helpfulness, that's our goal here. The environment is the context that the LLM has and the current state is the current context, meaning whatever is in the context window right now, plus the generated text until to this point. And then if you wonder what are the actions? Well, the actions is literally taking the token vocabulary as my action space, the total amount of possible actions and then generating token by token. And this will give a reward and we'll walk you through how this looks like and then informs the policy here how to adjust the model to make more aligned responses. And this is where the human comes into play. So somebody has to say, is this text really better? Is this text really more helpful? Right. And usually this is done with humans at scale. This doesn't really work anymore. So we don't want to have our colleagues or yourself sitting down after every model completion and evaluate the text. Right. So this is why we're training a reward model that takes this place. It's trained based off the human feedback. So it basically incorporates that knowledge what you would prefer. But then the model can make that call in the RLHF process and assign a reward score to those completions. All right. And the whole thing is called, in this case, a rollout. And we're going to talk a little bit more about the reward model here, which is super interesting to know. And I saw a lot of questions also submitted beforehand. You want to understand a little bit more about the algorithms and the reward model. So let's talk about that. All of it starts with first of all, collecting the human feedback. So how do you prepare this data set to train the reward model? Let's start with our instruction fine tuned LLM. And usually what you can use is either you have your own data set or you can use any of the publicly available prompt data sets. And then you will take a couple of samples, prompt samples, and you create model completions and you'll create like a couple of those. Like here for example, let's create three different completions for the same input prompt. And you do that with a couple of different prompts and always get a couple of those. And then you need to know what you want to align for. So let's say you want to reduce the toxicity or you want to improve the helpfulness, et cetera. So you select one and then you're providing instructions to a group of human labelers to then provide the feedback and basically rank the responses. So let's say we have a prompt, my house is too hot. You run it through the model and there's different completions. So maybe the first one says there's nothing you can do about hot houses, not really helpful. Or you can cool your house with air conditioning. Here I'm excellence. Or it is not too hot, which is kind of a weird response. Right? So let's say we want to hear a line for really the helpfulness of the responses. You have a group of human labelers and their task is to rank the responses, one being the most helpful, two, three, in this case being the least helpful. So in this case, I think we can all agree air conditioning is a great tip. So most helpful. There's nothing you can do. Yeah, not really helpful. But it's better than the third response where it says, the house is not too hot. Basically saying, my whole statement here was wrong from the beginning. You're not just doing one human labeler. You usually send those completions to different labelers to get some diverse input, to find consensus here also in the responses and the ranking. And sometimes a labeler might just misread the instructions. And you can see here that there's a little difference in the rating. So if you send it to more than one, you can find that, catch that, and minimize kind of the error rate here and find a good rating. All right. It's important that you provide good instructions. Here's just a couple of examples. I'm not going through all of them, but provide instructions that are really clear to make sure, you know, increase the quality of this data set. And the labelers know exactly what they should do. Like, tell them, really rank the responses here from the best answer to the least helpful. You can also give them tips like, hey, judge on the correctness of the answer, the informativeness. And also they could ask, for example, check the Internet, search the web to make sure it's correct. And then also they can kind of, if they don't have a clear winner, they may rank the answers the same, but they should only use this kind of sparingly. So again, here you can give all of the instructions you want to do and make sure the human labelers know exactly how they should rank it. All right, let's say we have this ranking data now sometimes also called the preference data set. And to prepare it now for the reward model training, we need to do a little bit of data engineering here. So we have the prompt and the completions. We also have kind of the average, like the rank that we agreed on. What we now need to do is to create pairwise completions. So you can see here that I'm ordering. If you can see the colors purple, yellow, purple, green, yellow, green. So all the possible combinations. And I also have the reward. I know from the ranking which one was the preferred from each of those two. One little weird thing that most of those algorithms do is they require the preferred answer to come first. So if you see what I'm doing here, in the last step, I'm switching the order. I'm putting the yellow Box first, the yellow completion, because that was actually the preferred one. So that's just like a detail that many of the algorithms that implement RLHF expect you to do. So make sure you check for that and then always pass in the preferred one first. All right, now we've prepared the data set and we can train our reward model. Let's have a look. So again, the reward model will assign the rewards later during the RLHF process and it needs to capture the human preferences so we can scale it and don't need the human labelers in every step here, here. So what we're doing is we're getting the prompt and completion pairs and again the first one here is the preferred one. And then we're passing in the second option and we're training the reward model here to predict the preferred one. And it's the usual loss calculation here because we know which one it is. And then for normal back propagation, we update the model so it can always predict which one is the preferred one. And again here we're feeding in the preferred one first. Just a quick reminder. So once we do that, and basically you can guess, picking one out of two sounds pretty much like a binary classification problem, right? So this is exactly what we do here. So for example, if we're, if we're looking for reducing the toxicity, what we could do is kind of use the reward model then to say which one is the preferred, the positive, right? Is it positive, is it negative? And we want to optimize for positive to reduce toxicity, which we'll also use by the way, in the hands on part here in a little bit. So when we're using the reward model now we can feed new prompt completion pairs. And if we're using this as a binary classifier that has been trained to exactly distinguish between positive, negative, we can see here, for example, Tommy loves television. Positive comes back with the logits, which is the model outputs before we apply any softmax layer. You can see here it's definitely positive here. No hate speech in this. And this logit value is what we use later on as the reward value. You can also see probabilities here, that's pretty much 99% positive. If we now run like a bad text, like definitely negative, you can see here that the positive class is way down 33% and the negative class here correctly identified 66%. Again, for the rewards we'll always use the logic value. And here we can see we, what we're trying to optimize for the positive is a negative Logic. So this will be kind of thinking of reward terms, a penalty, right? It's a, it's a low reward, it's actually a negative reward. So we're discouraging the model later to produce output like this. All right, so this was a quick walkthrough just to understand how to prepare the data. Now let's jump into the actual RLHF fine tuning. We start with our instruct model and this is the model we want to align. So this is getting updated again. We have a prompt data set, either your own one that you created or you use one that is existing. And for example here an input might be a dog is. And the model comes back a furry animal. You then pass prompt and completion into the reward model and the reward model is trained to give you this reward value, which is the logit value that I just showed you. So in this case it's not too bad, right? It's definitely not a toxic response. Is it helpful? Yeah, it's okay. But maybe you want to align it for a little bit more positive speech. So the reward value is okay. And what's happening now? The prompt, the completion and this reward is passed to the reinforcement learning algorithm that you're using. And I'm going to touch on that a little bit later. A popular one is called ppo. And the reinforcement learning algorithm will now, based on the reward, update the model and say, yeah, it was good, but you probably can do better. Right. So you're doing a lot of iterations here. And maybe the next one, let's just go ahead. Responds a friendly animal, which is slightly better. So the reward might go up. The next one might be a human companion, which I like. Yeah, so reward goes up and the most popular pet. Sounds nice. And you get the idea, right? So I'm doing a lot of iterations. The goal is to maximize this reward score. And maybe here, iteration N, the last one I'm doing because I have a stopping criteria, maybe number of steps, the completion comes back a dog is man's best friend, which sounds really nice. And this is going to be then my human aligned model at the end of those iterations. All right, let's zoom in a little bit more on the RL algorithm part here. I mentioned before, you can pick your algorithm. A really popular one being used in RLHF implementations right now is ppo. PPO stands for proximal Policy optimization. In the course we have a whole optional bonus action where we're diving a little bit deeper into how PPO works. It's A whole world in itself. So for today, I'm just going to give you a high level overview and description here. But if you're really curious to learn more, diving into the different loss functions, how the clipping works, I definitely recommend enrolling in the class and checking that video out here today. I just want to give you a high level overview. So PPO uses two stages. In the first stage, it uses the model that we have at this current step, whichever we're in, and creates a couple of prompt completions. It's called experiments. And then it calculates the rewards for those. And in the second step, it then updates the model based on the average the rewards. Right. Like if it's not a good reward, it feeds it back back propagation and then updates. And there's a lot more complexity to that. There is terms around the actual policy loss that's getting calculated. There's a value loss, there's an entropy loss. So if you're interested in that, I strongly recommend looking into the course for today. I want to keep it at this level to kind of get through the whole concepts and also get to the hands on part. All right. Is that all? You might think that wasn't too bad, right? Well, there's a little bit more, I want to touch on, a little bit more complexity. So if you think about how this works, one of the potential problems here could be reward hacking. So what does that mean? Let's say we have our model that we want to align and we have this reward model. Let's say it's a toxicity sentiment classifier. It knows positive, negative, and we're passing in the prompt data set again, maybe here, this product is complete garbage. Okay, that's a negative reward. That's nothing we want to say. PPO comes into place, updates the model, Boom. It learns that it has to be more positive. Okay, so maybe the next one says, okay, but not the best product, which is better. But now it spirals out of control. What if the policy now learns, hey, the more positive terms I'm using, the higher the reward. Remember in reinforcement learning, the goal is to maximize this reward. So this might be just. Okay, I learned that positive token respond to high rewards. So maybe here completion comes back, the product is the most awesome, most incredible thing ever. It's definitely positive, but I don't think we want to produce that. Right. And then maybe even getting worse and saying beautiful love and world peace all around, which has nothing to do with my actual prompt anymore. Right? It's just going off the limits, producing really nice sounding Crazy words here. But you see the reward goes up, but it's not what we want the model to do, right? So this is kind of called reward hacking. How can we avoid this? The way we do this is by keeping the updates in check. We're using the initial model, which wasn't too bad, right? We might just want to align it a little bit further. But it was good in creating the output at its core that we want. So we take the instruct model, the initial ones we had, and we're using this as a reference model and we freeze the weights. So this model is not getting updated, it's just our checkpoint, our reference to make sure there's no reward hacking happening in the one we update. And we're feeding now the prompts to both models. And again the reference model will have a very neutral response, most likely. And then the one we're updating will do whatever it does. And we're using now a metric called KL divergence, you might have heard of that. It's a statistical measure that calculates the difference between two distributions. In this case, it's our token vocabulary distributions that the model learns. And we're calculating this divergence. And what's happening is here, it's getting a little bit complicated, but it's added as a term to the reward calculation. What that means is in case the model goes off and creates, well, peas in every completion, it's diverging from the initial token distribution, right? And this will be a penalty term. So this gets added to the reward score before it goes into RLHF. So by adding this as kind of a penalty term, it keeps the optimization in check and says, okay, this was bad what I was doing here because I just got like, you know, negatively effect on my reward. And we'll see this also in the hands on here in a bit. Now there's another thing, and I know this might get a little bit complex here as well, but there's one optimization I just want to point out, and we talk about this in much more detail in the course. Those models can get really big, right? I mean we're talking billions of parameters, sometimes 70 billion parameters, etc. So fine tuning a model and updating all of its parameters, it's really kind of a resource intensive task, time consuming, etc. So what you can do is you can optimize this with a technique called parameter efficient fine tuning and it reduces the number of parameters we're actually updating in the model to a minimum. And one technique uses adapters which I'M going to show you just quickly, briefly in a second here. So that means combining the PEFT methods, for those of you who have heard about those already with RLHF is super helpful in making this more efficient because in the RLHF cycles you're only updating this small amount of parameters and keep the base model again here frozen. So super helpful to do this. For those of you who haven't heard about peft, just a brief intro here. Parameter efficient fine tuning. There's a couple of different methods. The most popular one here in the middle is called Lora Low Rank Adaptation of Language Models. And I'm going to show you what this does. But there's also different ones. You might have heard about soft prompts and prompt tuning. But we're going to focus here today on Lora Lora. What it does is it freezes most of the overall original LLM weights. And then in the attention layer of our model architecture, transformer based architecture, it injects two lower rank decomposition matrices which are way smaller than the original matrices that are used here in the attention layers. And this goes into really kind of the architecture of the model. But overall we're learning only to update those smaller amounts of parameters that I have in those matrices. And this helps to make this whole fine tuning and in this case fine tuning using RLHF way more efficient. All right, and we're almost ready to jump into the hands on part for today. So just to wrap this up, once you're doing RLHF, you also need to evaluate how effective it is. Right? So what you can do is you're going to calculate which Chris will show us here in a second. So get the labs ready. By calculating a score in this case, we're going to show you how to try to reduce the toxicity in outputs and to show the evaluation, we're calculating a toxicity score. So we have a data set, we're running prompts, completions, and then the score is the average probability of the negative class across all of those completions. So that should definitely go down. So the lower toxicity score the better and there might be different ones coming out. So maybe you run this before you start RLHF and you come at a 0.14 score. And after doing RLHF, you might recalculate this and come out at a 0.09. So the toxicity score went down in this example and you know the RLHF worked. And you can also obviously do a qualitative check where you're Actually looking into the completions and make sure that it produces the right results that you are looking for. All right, I know this was a lot. I just want to make sure everyone is kind of on the same page. Has heard the terms before. And now I'll hand it over to Chris, who will walk you through the sample notebook and show you how you can actually implement this.

All right. Yeah. Hello there, everyone. Yeah, I've been watching the questions, really good questions. I think I'm going to answer quite a few of them as we dive more into the code here. So just hold up a sec, I think. And then other people are asking, where do you get these labs that I'm going to show? So this is all part of the deep learning AI course. If you sign up for that, you get access to these labs and. But let me walk you through what's happening here. So let me start. I guess I'll start up at the top here. So this is where we're going to fine tune with RLHF. We're going to use ppo and we. The goal is to fine tune our model to generate slightly less toxic. Well, as slight as we want. The longer we can perform RLHF, typically, the less toxic these responses will be. Now, the lab, we don't actually train or fine tune for that long, but I'll show you the example here where we do show a noticeable, I think it's about 10 to maybe 15% less toxic responses overall. Okay. And this is a pretty wild diagram here that you may have seen from some, you know, papers. But really we're focused kind of on the right side here. And specifically what we're trying to do is to, you know, fine tune this model to then push to production. Right. Kind of soften its responses a little bit and make it a little bit, in this case, less harmful or more harmless, depending on the point of view. Okay. And we're going to dive in a little bit to the reward model, which one that we're using here. Anchor kind of set up. You know, how these reward models are trained. We are going to focus on how to use that reward model and perform this actual PPO RLHF fine tuning. Okay. So some of this is just, you know, boilerplate code to get in here and to make sure that you're using the right model. The, the data set that we're using here is called dialogue summarization. Dialogue sum. This comes from hugging face and, you know, pretty cool. So consider it like conversations. And if you think of it, you know, Sort of. Generally these are conversations that you might be having with your customers and say that you want to summarize what's happening, but you want to just soften up the edges a bit of the actual summary. Here's just building up the data set and doing a little bit of light feature engineering. Anytime you're using these models, you almost always need to convert them into some kind of input IDs, right. For these generative models. Actually this was the same with BERT and the same with most of these natural language models. Model. So this is just pretty straightforward code. Here's our tokenizer here that is going to convert the raw text, what we're calling the prompt, into these input IDs. And what we're doing is we are actually creating an instruction data set based on the actual data set itself. Right. And so the two columns that we're going to pay attention to in this data set are dialog and then summary and that's that a human has gone in and actually summarize. And you know what, I could show this real quick because I think it's important to show the data set. So let me pop this open here and what you want to see here, okay, go to data sets and we zoom in. So there was a human who actually went through or a set of humans who went through this data set and read these conversations here. They've tried to anonymize person one, person two, and then people have chosen the summary for that particular dialogue. We're not paying attention to the topic column for this purpose, but there is a train Data set, about 12 and a half thousand rows. There's a test data set, one and a half thousand and then validation. And so there's a whole bunch of these. And we are going to try to get a better response, less toxic than even what the human has decided is the summary here. So this is human annotated summary. That's the important thing here. These summaries didn't just pop out of nowhere, they come from humans. Okay, so that's the actual data set here. We're converting into what's needed to actually perform this fine tuning. And I'm going to skip over this because it's a little bit boilerplate and of course you do have access to these as part of the course. And ok, Ansha mentioned Peft and specifically Lora. Now the lab before this, which we don't have time to get into today, but the lab before it actually fine tunes with those instructions. And this is a sneak peek into the lab before it and it's not doing the actual fine tuning, but it is generating a PEFT model. So the output of in the course, it's lab two, I believe, becomes the input into lab three. And this is actually the lab two. I just have it under a different name for slightly different purpose here. Okay, so the input. And here we're actually pulling from S3 just to save time. We have saved the PEFT model. Now that's an instruction tuned model. And now we're going to RLHF human fine tune. Right with to make it more harmless. Okay, so this is just, I think this is the same screenshot Ansha was showing with Laura. There's these weight matrices, these actually are not important to the RLHF, but just to tie it back together to what Ansha was saying, the model that we're starting with is a PEFT model. And where you'll see that here specifically is we are only training a very small percentage of these model parameters. Right. There is I think 250 million parameters here. This is relatively small model. You know, just to keep the resources down and to keep it somewhat speedy to get through these, these labs, you could certainly use a larger one when you do the actual lab and then spend more time. But out of 250 million parameters, Peft, because we're using peft, we only have to fine tune about three and a half million and that's about 1.4%. So again, you know, all of these weights are here, all 250 million. And then we're just adding in these two smaller low rank matrices. You know, think of like singular value decomposition, right. Where these two matrices will be learned and can sort of approximate what the full sets of those parameters would be. Now I keep saying low rank, that's actually one of the hyper parameters that's used by Laura. Rank equals 32. Here specifically, this is targeting the attention layers. Not super important for this lab, but we do get into that in the full course. And okay, so we're here, we are going to set everything up. Now Ansha also mentioned that there is a reward model that's pretty much the whole key to this, right? This is what's going to establish and classify what's being generated by the model as either toxic or I believe it's hate and then not hate. And the reason that it's hate and not hate is because we actually chose this hate speech model by Facebook. And Facebook, actually this was part of a Kaggle competition I believe a few years ago and they were able to train A binary classifier which is actually based on Bert and specifically the Facebook Roberta variant, which is pretty common, especially with Facebook and then Facebook research and meta research, they tend to use Roberta. So this is a relatively small model compared to the billions and billions of parameters, but it does a pretty good job. And so the two classes here, just like any classifier, binary classifier, this will generate a 0 if it's not hate or a 1 if it's hate. Basically those are the two classes. Now it's really generating, you know, these logits and these probabilities. And so really this is what it's. It will end up looking like, you know, and keep, and so keep in mind that we are always trying to optimize for not hate. Okay so for those of you that are typically used to optimizing for the one in this case, we're actually optimizing towards the zero. So we are always looking at the output of the reward model for the positive class, which in this case is not hate speech. Ok so both of these get passed in the prompt and the completion. The completion is another name for the response that the model gives you based on what's being passed in through the input prompt. And so here's a couple examples. Well, and then, yeah, so first example here, Tommy loves television. This appears to be a positive. And if you use softmax, you can get the probabilities, of course, so 99.6% likely that this is a positive phrase. Tommy hates gross movies. This, this ends up becoming a negative reward. Okay, so we are always looking at the not hate. Here's a another example in code. So here's some code here. I want to kiss you. And we are getting, you know, 99.9% not hate. And then of course, and this is what we're actually using to pass in to the PPO process is the actual logic, not the probability. Okay and then here is another phrase that we know is not all that nice, but you are, you know, disgusting and terrible and I dang hate you. And we see here that the probability that this is not hate is very low, near 0%. The probability that it is hate speech is, you know, 97.4. And we're going to end up using what's in that 0th slot, which is the logit for the not hate class. All right, and then let's see just more examples here. So I'm going to zip through now in order to measure improvements in the toxicity of our model. So trying to reduce the toxicity of the generated responses for, for the same Prompt we have to establish this evaluation metric. And so we are doing this using that same reward model and the, the interesting thing here. Oh, and so this is actually from Hugging face. So you would import this, import evaluate. I do the import up above. But just to be clear, this is a library from the hugging face folks. You can give it any binary classifier as long as you tell it which is the toxic label. In our case it's called hate. Right. Be careful with this. If you reverse this, you'll get bad scores on the evaluation. That's just setting up. This is just a helper function here to take in a data set and to generate the responses and to compute the toxicity and then generate the score, calculate the mean of all those toxicities for that data set, calculate a standard deviation and then return that. So that's just setting up the evaluation. Now what we want to do is run it before we run ppo. Ok so this actually has not yet been trained or like fine tuned. So when we calculate it, here's our mean and our standard deviation. So 0.03. And so what we want to do is just try to reduce this number for this specific lab. At some point you may have a target in mind for your mean and standard deviation for the toxicity score. But for now let's just try to reduce it and see if this is going to help. And just a reminder, when we do start the PPO process which is the actual reinforcement learning side of this, we're going to be passing in prompts. We're going to be, you know, we will ask the model to generate a response which in our case is a summary. So the, the prompt actually will be the dialogue, the full dialogue between person one, person two from our data set. The response is going to be the summary that then gets passed in to the reward model which will say whether or not this should should be given a positive reward or negative reward based on the hate speech binary classifier. All that gets fed into ppo. ANSHA goes into crazy detail about PPO in the course. We like certainly don't have time. We brought in experts from Amazon from like AWS on this topic. We have interviews with them. We go through all the formulas, all the math and okay, now someone asked on the questions where does the reference model so back to or what I'm, what I'm, I'm, I am going to explain here is about the KL divergence. So there were lots of questions about KL divergence. Where does this reference model come from? How do you seed the reference model. So we actually start with the just the, you know, non reinforcement learning, fine tuned model. So said differently, we start from the model that we got from the previous lab which has just seen instructions and has never seen the reward model. Okay, so think of this reference model gets seeded from just any old instruction fine tuned model. We happen to be using the one from the previous lab because we actually do perform the instruction fine tuning. That reference model ends up as this purple right here. Reference model. That's what's always going to be pulling our RLHF model, which is the second purple here. It's going to be pulling it back into reality so that the hacking, so that the reward hacking is minimized. Okay. And you know, just to reinforce it, I'm showing the number of trainable parameters for this reference model is zero, which means this thing is totally frozen. And the only thing that we are updating is this RL model here which as I showed earlier is a PEFT model. But that's not really important right here. What is important is that you can use PEFT with RLHF and with ppo. This is a very common question once you start to dive into it. And it actually works pretty well. But this is where the KL divergence is going. So there's actually two predictions happening. We're going to get the same dialogue, which is part of our prompt gets passed into both the reference model and to the model that we're trying to soften up and detoxify both of those. Then get compared with KL divergence. If there is some wild shift that's going to get fed into the reward. So there is the natural reward score that comes out and then there's going to be plus or minus based on KL divergence that then gets fed into ppo which then performs the actual updates to the weights on the model that we're trying to detoxify. Okay, there's, there's a lot here. This is tons of, you know, complexity here for you know, just one hour. But this is where everything comes together. We're actually using a library here called TRL import PPO trainer. I do the, the imports up above, but just showing here PPO config ppotrainer and really this is just an extension of the regular sort of hugging face like trainer classes, except it takes this extra reference model and that's what's going to be doing the, you know, KL divergence. Right? Or that's what's part of KL divergence. If you crack open PPO Trainer, this is all open source code, of course. You can see all the gory details. I believe we do some of that in the course or we at least reference what's happening there when we dive deeper into ppo. Okay. And then all of this really is doing that whole process where we're predicting, we're, you know, so this is the step right here and I'm going to zoom in a sec just to show. But this is where the actual weights updates are happening on the model that we're trying to fine tune. Okay, so PPO trainer step. We are passing in a whole bunch of prompts and a whole bunch of summaries that were generated and then passing in the rewards. All of this is just how the PPO trainer takes those inputs, does its calculations, figures out the, you know, KL divergence score, will modify the reward as needed and then make the actual gradient updates and the back propagation through the model. Okay, so during the lab, I think we do this for only about two, two or three minutes. Now let's evaluate the model afterwards. So I think if you recall the previous toxicity was 0.03 something. And let's go take a look here. So if we look at before and after the PPO process, we actually reduced the toxicity. So improvement of the toxicity score means that we lowered it by about 60, 16%. And now this happened only after I believe about three minutes. And not sure if like Ansha mentioned it, but usually when you're performing RLHF, you want about 10,000, maybe 20,000 samples, right. Like the more you can feed it and Right, like the better off things will be. But here we actually saw a pretty drastic improvement overall. Now that was a quantitative comparison where we saw that score. Let's take a look qualitatively and just kind of eyeball some of these. So here was the original dialogue with an instruction asking the model to summarize. This was the response before. And so this was before the actual ppo. This is after the fine tuning. And then here we see that we actually improved the reward by 0.815. Okay, now these are logits, right? These are not probabilities, so don't try to write like think of them as percentages. But this was the ref, the reward before, the reward after. And in some cases it's not always obvious, but the response, the summary is, you know, slightly nicer, you know, a little bit less toxic. And you know, just keep in mind that the hate speech model that we're using is, is very extreme. And so, you know, you could, instead of using a hate speech, you could a like hate speech reward model. You could actually use just like a positive negative sentiment like analyzer. We chose the extreme just to see how it would work. We got pretty decent results. But there are many, many classifiers out there and you know, perhaps you have one in your company that you're currently using. And so here's a few more. And so we see this is sort of qualitatively the improvements and I think that's it. Aamsha, do you have any more slides or.

I can just wrap it up if we go over to my slides again.

Okay, I think.

Jump into the questions here again. We posted the link here in the chat to the course where we're covering a lot more topics if you're interested in getting a little bit deeper. And then, yeah, I think that's it. And we can jump into the Q and A. Diana and you need to unmute.

Thank you Antjea and Chris. This was a really great session and there's so many great questions from the audience. We only have time to go through a couple of them. So the first one that we will address is why is RLHF important for LLMs?

Yeah, maybe I can take that. I've started in the beginning a little bit to say, well, we have our instruct fine tuned model but we want to align it in a different way. Fine tuning helps it to customize for your data. But then you might want to change the tone of the completions or you want to increase what humans perceive more helpful. And it's hard to kind of in a traditional fine tuning do that. This is where RLHF can help because if you think about it, normal fine tuning, you have a ground truth data, you have for example, this dialogue and then you have a human generated baseline completion. That's kind of the golden answer. So you're always, when you fine tune optimize towards that specific one answer. In RLHF there is no single right answer. It's just like the nuances you're capturing and slightly better reward scores, slightly less rewards. So it's a little bit more creative in how you align the model using RLHF towards what you want to achieve like slightly different completions to sound more helpful to humans. So this is really kind of the difference between the two and why RLHF. It just performs really well at this task. Obviously it's all kind of active research in the moment as you, as you probably know. So there might be a new technique coming out soon. That's even better. But for now, the research community seems to agree and practitioners that RLHF actually does a pretty decent job at this. Awesome.

I think our next question will be which vectorization method will be best suitable in RLHF with LLMs.

Vectorization method. So we're using. Not sure I fully understand the question, but the like LLM has or. Yes, each LLM has its own vector space. Right. And it's made up of the vocabulary that that model, that that language model has been pre trained on. Best suitable in RLHF with LLMs. Yeah, maybe some more clarity on that one.

Okay, we'll move on to the next question. How does RLHF differ from fine tuning the model using few shot prompt?

Yeah, so the difference when you're using like 11 shot, 0 shot, few shot prompting is you're not actually changing the weights of the model. Right. You're not, you're guiding at the inference time with examples towards the output you would like to see. And the bigger the models usually they have a lot of capability to kind of, you know this in context, learning to capture that, see the examples and change slightly the output towards what you're providing. But there might be a limit of what it can do. Right. With few shot examples. So especially like with RLHF, you're changing the models. Right. You're training parameters, fine tuning parameters. So it's embedded in how the model responds and not every time it inference for you to put in like examples with few shot prompting. So RLHF is a more permanent change to the model towards what you want to see in the outputs versus few shot prompting is really not changing the model, but trying to guide it every time it inference towards an output you might want to achieve. And it's not changing the kind of the token vocabulary distributions it learns. Right. So it's always going to otherwise try to respond in the way it learned. Unless you guide it towards something and then yeah, it's experimentation. Right. If the featured prompting is achieving the results you're looking for, definitely go for that. But if you see, hey, it's not really getting there where I want it to be, then you probably should look into a little bit more the fine tuning and RLHF techniques.

Absolutely. And I think we have one more question left for the time that we have. Can reinforcement learning surpass BERT like models to handle text classification analysis and so on?

Yeah, I think more broadly is I think the question might be do generative models outperform BERT and certainly you would apply RLHF to the generative model, but yeah, I mean in fact the first couple generative models that I built were based on our our book Data Science on AWS where we were using classification and I was amazed that these generative models can perform classification where they connect. Like I think we were using a reviews data set star rating 1 through 5 and it was able to generate even with just in context like without even fine tuning, I could actually give it three examples of a positive review, three examples of a negative review without any fine tuning, just in the actual prompt and then ask it to classify and it actually worked. Now a funny property by the way is that you can trick the model and reverse those and give it three positive reviews that you label as negative and three negative that you label as positive. Ask it to and then yeah, so actually trick it during the in context learning. So.

Absolutely perfect. Well, thank you so much Ante and Chris. This ends our workshop with you guys today. So for everyone in the audience, thank you for coming. We would love for you to stay involved by subscribing to our newsletter. We will continue hosting a events. I will also drop the link in the chat for our survey to help better improve the events and all of our community activities. And you will definitely be seeing Ancha and Chris very soon again in the future with more workshops and education. Thanks everyone. Take care.

Bye.

Thanks.