OpenAI’s ChatGPT Is Now Learning From Another AI!

Today we are going to marvel at a fantastic paper that tells us how to create a chatgpt like AI assistant and what is under the hood and how to do it better. Spoilers. OpenAI just found a way to use an AI to train another AI. It's crazy, I know. Let me try to explain. ChatGPT like AI assistants typically consist of two puzzle pieces. Piece number one is the knowledge base. This is a neural network that has read a ton of data. Internet forums, textbooks, school exams, film reviews, you name it. From this data, it is able to learn most languages on the planet and it knows about important movies, events, and of course, it is an excellent scientist. Let's call this neural network GPT. However, it is not the assistant that you can use in ChatGPT. No, no. @ this point, all it can do is take a sentence and not answer it. Not even close. All it can do is complete it. Yes, that is not very useful. We usually don't want our questions completed, we want them answered. How does a knowledge base become a an assistant? Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Now comes the second step, coaching. We need to coach it. Or in other words, teach it how to be useful. Believe it or not, this looks like playing a video game. Oh yes. We give it example questions, it provides a bunch of answers and we ask humans to assign scores to these answers. Better answers get a higher score. This will look like playing a video game where it tries to maximize the score. But to do that, you don't need to explore or defeat enemies. Instead you need to answer correctly. From this, it will learn to be not too vague and not too specific, and to not just answer every complex question with, well, if it depends, and not say anything else that drives us humans up the wall. And now, hold on to your papers, fellow scholars, because scientists at OpenAI had a crazy idea in their new paper and that was to use an AI to teach this AI instead. Not for everything, but for safety related questions. That is very important. So teach an AI to teach another AI. Yes, that is exactly the case. So does it work? Well, it does. The chatgpt that you fellow scholars are using day by day is already made that way. I found this to be very surprising. You see, not every prompt should be answered. If we ask something really unsavory, the assistant has to make a decision, comply or refuse to help. If it refuses, it also has to provide an explanation of some kind. And what this paper proposes is to use an AI to decide when to do which. So how good Is it. Is it as good as humans? Hmm, what do you think? Can it be as good as a human? That sounds crazy, but they say that it is not only as good as a human in this, but even better. Wow. This is tricky to measure because creating a super safe AI is really, really simple. Just deny every request. Super safe. Also not very useful and thus they call these bad refusals. Ultimately, what scientists at OpenAI try to strike is a balance between safety and usefulness, and in this area it performs spectacularly. I find this to be a stunning result because it learned how to do this from humans and from very little training data. It is able to surpass the people who made the training data. The student outshines the teacher, at least on this small specialized task. It really helps when it is supposed to help and it refuses when when it should not be helping. And get this, it had 16% less bad refusals than humans do. Making these models more useful while keeping them safe. It's like teaching a little child how to behave and then this child could generalize this knowledge to new situations. This feels a bit like true intelligence to me. That is absolutely amazing. What a time to be alive. I would like to send a huge thank you for this because this is a paper that contains many theoretical and implementation details. So much so that they even give us the source code for this project, all of it free of charge. So this is the wondrous world of research papers. Subscribe if you wish to see more amazing papers like this. So what do you think? What would you fellow scholars use this for? Let me know in the comments below. If you're looking for an Nvidia GPU cloud service for large scale AI projects, check out One Click Clusters by Lamda, featuring up to 512 Nvidia H100 GPUs with InfiniBand networking and hold on to your papers because we with one click clusters you can access all these GPUs on demand with reservations starting at just two weeks. Try it out@LambdaLabs.com paper or click the link in the.