Andrew Ng and Nikita Namjafi teach a new course on RLHF, a technique that uses human ratings to align large language models with human values like helpfulness and honesty. You'll learn the theory and then fine-tune LLaMA 2 using real human feedback data. No reinforcement learning background needed.
I'm delighted to introduce Reinforcement Learning from Human Feedback, or rrhf. Built in partnership with Google Cloud and taught by Nikita Namjafi, RRHF has been a key technique for training large language models, or LLMs. After training a model on Internet data to follow instructions first by training on Internet data, then by a technique called instruction tuning, the LLM may still sometimes be impolite or can hand out harmful information. ROHF has been critical for aligning LLMs with human values to make them more helpful, honest and harmless. So ROHF is an important technique to learn. But even if you aren't training an LLM from scratch but are instead building applications, ROHF can still be very useful for aligning your application with values that you want. RHF is particularly effective for tasks where the desired output is difficult to explain or describe. For example, if you ask an LLM to listen about a personal problem, a legitimate response might be for the LLM to try to diagnose the problem and identify what you might do differently. But depending on your application, perhaps it's better for the LLM to reply with sympathetic words of encouragement instead. So how can we teach an LLM to better align with specific preferences such as this, which can be quite nuanced? RLHF provides a way for humans to guide the LLM by rating answers according to their preference. If you're familiar with fine tuning and reinforcement learning, I think that loosely supervised learning is to reinforcement learning as far as fine tuning of LLMs is to rohf. If you don't know the terms I just used, don't worry about it. It's not important. But the key is that sometimes it's more convenient not to have to give training examples as in supervised learning or fine tuning of an LLM, but instead to just give judgments or ratings about what you like and don't like as feedback for the algorithm to learn from. In this short course, you learn in detail how RLHF works and apply it to tune the LLAMA 2 model using data collected from human reviewers. I'm delighted to introduce our instructor, Nikita Nandrushi, who is a generative AI developer advocate for Google Cloud. She's helped a huge number of developers build generative AI applications with LLMs, and I look forward to her sharing her experience and insights with us here. Thanks, Andrew. In this course you're going to get an overview of the key concepts and theory behind the RLHF process. This will be a nice introduction to RLHF. Regardless of which tools you'll use to apply it. Then you'll get hands on tuning the Llama 2 model with RLHF. You'll get to walk through the entire workflow from preparing the data sets to tuning the LLM to evaluating the results. And you don't need to have any prior knowledge about reinforcement learning. And so I hope you will take this course to reinforce your learning in generative AI. Definitely. Thanks Andrew and I hope you all enjoy this course.
