OpenAI trained a summarization model on 100,000 Reddit posts with TL;DRs using reinforcement learning from human feedback. The model matched or beat human-written summaries and generalized to news articles, sometimes outperforming its human teachers.
Questions this video answers
March 30, 2021
About 100,000 well-curated Reddit posts
A simple baseline that takes the first three sentences of an article
Reinforcement learning from human feedback (RLHF)
Chapters
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. This paper will not have the visual fireworks that you see in many of our videos. Oftentimes you get ice cream for the eyes, but today you'll get an ice cream for the mind. And when I read this paper I almost fell off the chair. And I think this work teaches us important lessons and I hope you will appreciate them too. So with that, let's talk about AIs and dealing with text this research field is improving at an incredible pace. For instance, four years ago in 2017, scientists at OpenAI embarked on an AI project where they wanted to show a neural network a bunch of Amazon product reviews and wanted to teach it to be able to generate new ones or continue a review when given one. Upon closer inspection, they noticed that the neural network has built up a knowledge of not only language, but also learned that it needs to create a state of the art sentiment detector as well. This means that the AI recognized that in order to be able to continue a review, it needs to be able to understand English and efficiently detect whether the review seems positive or negative. This new work is about text summarization, and it really is something else. If you read Reddit, the popular online discussion website, and encounter a longer post, you may find a short summary a TL doctor of the same post written by a fellow human. This is good for not only the other readers who are in a hurry, but it is less obvious that it is also good for something else. And now hold on to your papers because these summaries also provide fertile grounds for a learning algorithm to read a piece of long text and its short summary and learn how the two relate to each other. This means that it can be used as training data and can be fed to a learning algorithm. Yum. And the point is that if we give enough of these pairs to these learning algorithms, they will learn to summarize other Reddit posts. So let's see how well it performs. First, this method Learned on about 100,000 well curated Reddit posts and was also tested on other posts that it hadn't seen before. It was asked to summarize this post from the Relationship Advice subreddit and let's see how well it did. If you feel like reading the text, you can pause the video here. Or if you feel like embracing the TL Dr. Spiritual, just carry on and look at these two summarizations. One of these is written by a human and the other one by this new summarization technique. Do you know which is which? Please stop the video and let me know in the comments below. Thank you. So this was written by a human and this by the new AI. And while of course this is subjective, I would say that the AI written one feels at the very least as good as the human summary. And I can't wait to have a look at the Moore principled evaluation in the paper. Let's see, the higher we go here, the higher the probability of a human favoring the AI written summary to a human written one. And we have smaller AI models on the left and bigger ones to the right. This is the 50% reference line. Below it, people tend to favor the human's version. And if it can get above the 50% line, the AI does a better job than human written TLDRs in the dataset. Here are two proposed models. This one significantly outperforms and this other one is a better match. However. Whoa, look at that. The authors also propose a human feedback model that even for the smallest model, handily outperforms human written TLDRs. And as we grow the AI model, it gets even better than that. Now that's incredible. And this is when I almost fell off the chair when reading this paper. But we are not done yet. Not even close. Don't forget this AI was trained on Reddit and was also tested on Reddit. So our next question is of course, can it do anything else? How general is the knowledge that it gained? What if we give it a full news article from somewhere else outside of Reddit? Let's see how it performs. Hmm. Of course this is also subjective, but I would say both are quite good. The human written summary provides a little more information, while the AI written one captures the essence of the article and does it very concisely. Great job. So let's see the same graph for summarizing these articles outside of Reddit. I don't expect the AI to perform as well as with the Reddit posts as it is outside the comfort zone, but my goodness, this still performs nearly as well as humans. That means that it indeed derived general knowledge from a really narrow training set, which is absolutely amazing. Now, ironically, you see this Lead three technique dominating both humans and the AI. What could that be? Some unpublished super intelligent technique? Well, I will have to disappoint. This is not a super sophisticated technique, but a dead simple one. So simple that it is just taking the first three sentences of the article, which humans seem to prefer a great deal. But note that this simple lead tree technique only works for a narrow domain. While the AI has learned the English language, probably knows about sentiment and a lot of other things that can be used elsewhere. And now the two most impressive things from the paper in my opinion. 1. This is not a neural network but a reinforcement learning algorithm that learns from human feedback. A similar technique has been used by DeepMind and other research labs to play video games or control drones and it is really cool to see them excel in text summarization too. 2. It learned from humans, but derived so much knowledge from these scores that over time it outperformed its own teacher. And the teacher here is not human beings in general, but people who write TL doctor's along their posts on Reddit. That truly feels like something straight out of a science fiction movie. What a time to be alive now. Of course not even this technique is perfect. This human versus AI preference thing is just one way of measuring the quality of the summary. There are more sophisticated methods that involve coverage, coherence, accuracy and more. In some of these measurements the AI does not perform as well, but just imagine what this will be able to do. Two more papers down the line this episode has been supported by Weights and Biases. In this post they show you how OpenAI's prestigious robotics team uses their tool to teach a robot hand to dexterously manipulate a Rubik's cube. During my PhD studies I trained a ton of neural networks which were used in our experiments. However, over time the there was just too much data in our repositories and what I am looking for is not data but insight. And that's exactly how Weights and Biases helps you by organizing your experiments. It is used by more than 200 companies and research institutions including OpenAI, Toyota Research, GitHub and more. And get this Weights and Biases is free for all individuals, academics and open source projects. Make sure to visit them through wnb. Com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weightsand Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
