Dr. Zsolnai reviews Google Brain's Meena, a 2.6 billion parameter chatbot that scores near human level on sensibleness and specificity. DeepMind built a system solving abstract reasoning problems at 78% accuracy, while OpenAI's GPT-2 excels at sentence completion. Meena's performance metrics correlate strongly with human judgment.
Questions this video answers
Approximately 1.5 billion parameters.
2.6 billion parameters.
About 62% correct.
About 78% correct.
0.96.
Chapters
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. When I was growing up, IQ tests were created by humans to test the intelligence of other humans. If someone told me just 10 years ago that algorithms will create IQ tests to be taken by other algorithms, I wouldn't have believed a word of it yet. Just a year ago, scientists at DeepMind created a program that is able to generate a large amount of problems that test abstract reasoning capabilities. They are inspired by human IQ tests with all these questions about sizes, colors and progressions. They wrote their own neural network to take these tests, which performed remarkably well. How well exactly? In the presence of nasty distractor objects, it was able to find out the correct solution about 62% of the time. And. And if we remove these distractors, which I will note that are good at misdirecting humans too, the AI was correct 78% of the time. Awesome. But today we are capable of writing even more sophisticated learning algorithms that can even complete our sentences. Not so long ago, the OpenAI lab published GPT2, a technique that they unleashed to read the Internet and it learned our language by itself. A few episodes ago we gave it a spin and I almost fell out of the chair when I saw that it could finish my sentences. About fluid simulations in such a scholarly way that I think could easily fool a layperson. Have a look here and judge for yourself. This GPT2 technique was a neural network variant that was trained using one and a half billion parameters. At the risk of oversimplifying what that means, it roughly refers to the internal complexity of the networks, or in other words, how many weights and connections are there. And now the Google Brain team has released Mina, an open domain chatbot that uses 2.6 billion parameters and shows remarkable human like properties. The chatbot part means a piece of software or a machine that we can talk to. And the open domain part refers to the fact that we can try any topic, hotels, movies, the ocean, favorite movie characters, or pretty much anything we can think of and expect the bot to do well. So how do we know that it's really good? Well, let's try to evaluate it in two different ways. First, let's try the super fun but less scientific way. Or in other words, what we are already doing. Looking at chat logs, you see Meena writing on the left and the human being on the right, and it not only answers questions sensibly and coherently, but is even capable of cracking a joke. Of course, if you consider a pun to be a joke, that is. You see a selection of topics here, where the user talks with Mina about movies and the bot expresses the desire to see the Grand Budapest Hotel, which is indeed a very human like quality. It can also try to come up with a proper definition of philosophy. And now since we are scholars, we would also like to measure how human like this is in a more scientific manner as well. Now is a good time to hold on to your papers because this is measured by the sensibleness and specificity average score from now on ssa in short, in which humans are here, previous chatbots are down there, and Mina is right there close by, which means that it is easy to be confused for a real human that already sounds like science fiction. However, let's be a little nosy here and also ask how do we know if this SSA is any good in predicting what is human like and what isn't? Excellent question when measuring human likeness for these chatbots plugging in the SSA again the sensibleness and specificity average, we see that they correlate really strongly, which means that the two seem to measure very similar things. And in this case SSA can indeed be used as a proxy for human likeness. The coefficient of determination is 0.96. This is a several times stronger correlation than we can measure between the intelligence and the grades of a student, which is already a great correlation. This is a remarkable result. Now what we get out of this is that the SSA is much easier and precise to measure than human likeness and is hence used throughout the paper. So chatbots, eh? What are all these things useful for? Well, do you remember Google's technique that would automatically use an AI to talk to your callers and screen your calls or even make calls on your behalf when connected to a text to speech synthesizer, Something that Google already does amazingly well. Mina could really come alive in our daily lives soon. What a time to be alive. This episode has been supported by Lambda. If you're a researcher or a startup looking for cheap GPU compute to run these algorithms, check out Lambda GPU Cloud. I've talked about Lambda's GPU workstations in other videos and I'm happy to tell you that they are offering GPU cloud services as well. The Lambda GPU Cloud can train ImageNet to 93% accuracy for less than $19. Lambda's web based IDE lets you easily access your instance right in your browser and finally hold onto your papers because the Lambda GPU cloud costs less than half of AWS and Azure. Make sure to go to lambdalabs.compapers and and sign up for one of their amazing GPU instances today. Our thanks to Lamda for helping us make better videos for you. Thanks for watching and for your generous support, and I'll see you next time.
