OpenAI GPT-2: An Almost Too Good Text Generator!

Dear fellow scholars, this is two minute papers with Zsolnai. This is an incredible paper from OpenAI in which the goal is to teach an AI to read a piece of text and perform common natural language processing operations on it. For instance, answering questions, completing text reading comprehension, summarization, and more. And not only that, but additionally, the AI has to be able to perform these tasks with as little supervision as possible. This means that we seek to unleash the algorithm that they call GPT2 to read the Internet and learn the intricacies of our language by itself, to perform this, of course, we need a lot of training data. And here the AI reads 40 gigabytes of Internet text, which is 40 gigs of non binary plaintext data, which is a stupendously large amount of text. It is always hard to put these big numbers in context. So as an example, to train similar text completion algorithms, AI people typically reach out to a text file containing every significant work of Shakespeare himself. And this file is approximately five megabytes. So the 40 gigabytes basically means an amount of text that is 8,000 times the size of Shakespeare's works. That's a lot of text. And now let's have a look at how it fares with the text completion part. This part was written by a human. In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley in the Andes mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. And the AI continued the text the following way, quoting a short snippet of it. The scientists named the population after their distinctive horn, Ovid's Unicorn. These four horned silver white unicorns were previously unknown to science. Whoa. Now, note that this is clearly not perfect, if there is even such a thing as a perfect continuation. And it took 10 tries, which means that the algorithm was run 10 times and the best result was cherry picked and record it here. And despite all of these, this is a truly incredible result, especially given that the algorithm learns on its own. After giving it a piece of text, it can also answer questions in a quite competent manner. Worry not. Later in this video I will show you more of these examples and likely talk over them. So if you are curious, feel free to pause the video while you read the prompts and their completions. The validation part of the paper reveals that this method is able to achieve state of the art results on several language modeling tasks. And you can see here that we still shouldn't expect it to match a human in terms of reading comprehension. Which is the question answering test. More on that in a moment. So there are plenty of natural language processing algorithms out there that can perform some of these tasks. In fact, some articles already stated that there is not much new here. It's just the same problem, but stated in a more general manner and with more compute. Aha. It is not the first time that this happens. Remember our video by the name the Bitter Lesson? I've put a link to it in the video description, but in case you missed it, let me quote how Richard Sutton addressed this situation. The Bitter Lesson is based on the historical observations that one AI researchers have often tried to build knowledge into their agents, 2 this always helps in the short term and is personally satisfying to the researcher, but three in the long run it plateaus and even inhibits further progress and four breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness and often incompletely digested because it's success over a favored human centric approach. So what is the big lesson here? Why is GPT2 so interesting? Well, big lesson number one is this is one of the clearer cases of what the quote was talking about where we can do a whole lot given a lot of data and compute power and we don't need to insert too much additional knowledge into our algorithms. And lesson number two, as a result, this algorithm becomes quite general so it can perform more tasks than most other techniques. This is an amazing value proposition. I will also add that not every learning technique scales well when we add more compute. In fact, you can see here yourself that even GPT2 plateaus on the summarization task. Making sure that these learning algorithms scale well is a great contribution in and of itself and should not be taken for granted. There has been a fair bit of discussion on whether OpenAI should publish the entirety of this model. They opted to release a smaller part of the source code and noted that they are aware that the full model could be used for nefarious purposes. Why did they do this? What is the matter with everyone having an AI with a subhuman level reading comprehension? Well, so far we have only talked about quality, but another key part is quantity. And boy are these learning methods superhuman in terms of quantity. Just imagine that they can write articles with a chosen topic and sentiment all day long and much quicker than human beings. Also note that the blueprint of the algorithm is described in the paper and a top tier research group is expected to be able to reproduce it. So does one release the full source code and models or not? This is a quite difficult question. We need to keep publishing both papers and source code to advance science, but we also have to find new ways to do it in an ethical manner. This needs more discussion and would definitely be worthy of a conference style meeting or more. There is so much to talk about and so far we have really only scratched the surface. So make sure to have a look in the video description I left a link to the page paper and some more super interesting reading materials for you. Make sure to check them out. Also, just a quick comment on why this video came so late after the paper has appeared. Since there were a lot of feelings and intense discussion on whether the algorithm should be published or not, I was looking to wait until the dust settles and there is enough information out there to create a sufficiently informed video for you. This of course means that we are late to the party and missed out on a whole lot of views and revenue and but that's okay. In fact, that's what we'll keep doing going forward to make sure you get the highest quality information that I can provide. If you have enjoyed this episode and would like to help us, please consider supporting us on Patreon. Remember our motto, a dollar a month is almost nothing, but it keeps the papers coming and there are hundreds of papers on my reading list. As always, we are available through patreon.com twominutepapers and the link is also available in the video description. Thanks for watching and for your generous support and I'll see you next time.