OpenAI released GPT-2, a language model trained on 40 gigabytes of internet text that can complete scripts and technical writing with surprising coherence. A 750-million-parameter version is publicly available, half the size of the full model, with applications in code completion and text generation.
Questions this video answers
October 1, 2019.
About 40 gigabytes of Internet text (40 GB of plaintext).
About 750 million parameters (a half-size variant of the full model).
Danielle King and the Hugging Face company.
Weights & Biases; listed users: OpenAI, Toyota Research, Stanford, and Berkeley.
Chapters
Dear fellow scholars, this is two minute papers with OpenAI. GPT2 is a learning based technique which can perform common natural language processing operations. For instance, answering questions, completing text reading comprehension, summarization and more. What is absolutely amazing about this technique is that it is able to perform these tasks with as little supervision as possible. This means that we unleash the algorithm that they call GPT2 to read the Internet and it will learn our language by itself. To perform this of course we need a lot of training data. And here the AI reads 40 gigabytes of Internet text, which is 40 gigs of non binary plaintext data, which is a stupendously large amount of text. You see here an example of how it fares with text completion and the results were absolutely amazing. And not only that, because hold on to your papers, we can now try it ourselves and my results show that it can be as wise as a scholar and it can also cheat like nobody's business. So let's try something simple here. First, as everyone is talking about job security, I'd like to give it the start of this two minute paper script and see if I am about to be replaced. Let's see it continues the intro. The history of eastern Europe since 1989 has been an interesting and complex story spanning four decades. On one side you have a regime of western backed dictators from the mid-1990s and so on. Very scholarly indeed. However, I think I might be safe for now. And now something that I wanted to try since the inception of GPT2, which is feed it a snippet of something that I said about wavelet turbulence which is one of the best papers ever written and have the AI finish it. I tried to frame it as an exam question and was not disappointed by the results. So the text is there was a similar technique by the name wavelet turbulence which is. I hope you see what I did there. And look what it did. There was a similar technique by the name wavelet turbulence which is not at all what we are talking about, but I let you this decide whether it is worth your time to read it. Wow. GPT2 has been caught cheating red handed. I love it. Let's continue what it had to say. Anyway, my aim here is to describe how we use a fluid simulation to simulate smoke and liquid effects using just a few simple equations. Solving a few functions gives us a set of derivatives of the particle trajectories, I.e. equations that describe particles, how and where the particles will come and go across the liquid in the simulation. For simplicity, as a good teacher of high school students should know we can use a simple equation which simulates two particles that collide. I am stunned it recognized that we are talking about fluid simulations, which is already remarkable, but it went much further. The completion is not bad at all and is not only coherent on topic, but has quite a bit of truth to it. I will have to rethink my previous claim about my job security. The even crazier thing is that the size of this model is about 750 million parameters, which is only half of the size of the original full model, which is expected to be even better. I put a link to this website in the video description for your pleasure. Make sure to play with it. This is Mad Fun and GPT2 will also see so many applications that we cannot even fathom yet. For instance, here you can see that one can train it on many source code files on GitHub and it will be able to complete the code that we write on the fly. Now nobody should think of this as GPT2 writing programs for us. This is of course unlikely, however, it will ease the process for novice and expert users alike. If you have any other novel applications in mind, make sure to leave a comment below. For now. Bravo OpenAI and a big thank you for Danielle King and the Hugging Face Company for this super convenient public implementation. Let the experiments begin. This episode has been supported by Weights and Biases. Weights and Biases provides tools to track your experiments in your deep learning projects. It can save you a ton of time and money in these projects and is being used by OpenAI, Toyota Research, Stanford and Berkeley. It is very easy to set up. In fact, this blog post shows how we can use their framework to visualize our progress using XGBoost, a popular library for machine learning models. Get ready because this is quite possibly the shortest blog post that you have seen. Yep, that was basically it. I don't think it can get any easier. Make sure to Visit them through wendb.compapers, wandb.compapers or just click the link in the video description and sign up for a free demo today. Our thanks to Weights and Biases for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
