DeepMind's Flamingo model trained on over a billion images can caption photos, read text, and answer questions about videos. It matches or beats specialized models on video QA datasets despite using far less task-specific training data, though it still makes silly mistakes on unsupported inferences.
Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai. Today we are going to have a look at DeepMind's Flamingo AI that has looked at over 1 billion images and see what it had learned from it. And yes, we will also try to break it successfully. Now, its goal was to fuse AI models that understand language with other AI models that understand images. And what comes out? Absolute magic. For instance, it can be an amazing assistant. We can give it a picture, ask it what it depicts, and it says, a bowl of soup with a monster face. I particularly like this one because this image was also made by a different AI. So this is an AI commenting on another AI's work. How cool is that? Now we can also subject it to a test that is interesting because it is designed to break the brain of some humans. Look, we need to read the text and then name the color that was used to write it. If you try to do it quickly, it is easy to answer it incorrectly. And apparently this little AI is passing with flying colors. But it gets better. Look, it even knows that this is the Stroop test. And it also knows that humans read this kind of text slower. So let's pop the question. Is it challenging for you too little AI? It says, I am not affected by this difference. Whoa. It is flexing, but it backs it up. What a time to be alive. Now, what it can also do is something called few shot learning. What is that? Well, this is learning from only a few examples. Something that humans are really good at, but machines, not so much. For instance, we look at an image and say that this is a chinchilla. We look at another image and say that this is a Shiba. Then the third time we again look at an image and we say, this is so. Does it know what it is supposed to do with this? Remember, neural network based learning methods typically require thousands and thousands of training samples to learn something new. So can this new one do it? Yes. Oh boy. That is amazing. So this is few shot learning, and I have a hard time overstating how incredibly useful it is. For instance, we can quickly teach it to read handwritten text from a piece of paper. Or here comes my favorite. We can even ask it to reverse engineer the prompts that the AI used to create these images. We teach it on two examples where we tell it the piece of text that was used to generate it. And now for the third time, it knows the drill and does it itself. Or we can even ask for the ingredients or even the nutrients that are present in a meal or or what songs a soundtrack of a movie contain. And here is another reassuring thing. It does not fall for the old Apple plus iPod trick that many previous techniques fell for. So now let's try to answer three scholarly questions. Is it really better than previous techniques if we compare it to previous zero shot techniques? This is a challenging case where the AI has to perform something new and there is no contest. This new one is so much better. However, what I found even more interesting is that it can even give fine tuned AI's a run for their money too. This is insanity. So what is that? Fine tuned means an AI that was specifically designed and trained for one task and one task only. This new method can do many things at the same time. This is a generalist, so it is not expected to beat specialized techniques. How could it? And here comes the insanity part. It is not only competitive with these fine tuned techniques on several data sets, but wow. It even outperforms them on some of these despite having access to 10 and sometimes even a hundred times less training data. I am stunned. These improvements are concentrated around video based question answering datasets like NextQA and IVQA. Question 2. How does it improve with the number of shots? You remember the Flamingo example? We had two shots here. And the answer is that it gets it early. It does not need tens of examples to understand what we are trying to ask. Its zero shot performance is formidable, so it can do things it has never been taught before. That is incredible. But even in other cases, after just a few examples, it really gets it. And three, the usual model size. Yes, as I hoped, results still improve if we train a bigger neural network, which is great news. Two more papers down the line and it will be so much better even without algorithmic improvements. If we increase the model size, it gets smarter. And here is a beautiful chart that showcases that it can learn from only a few shots significantly better. Dear fellow scholars, this is perhaps artificial intelligence being born right before our eyes. What a time to be alive. But wait, we said that it is excellent at video based question answering. So let's push it to its limits and see what it can do. Wow, that is very nice. It understands that this good boy is being weighed. Understands what a video game avatar means, reads a sequence of text even if only a few letters are shown at a time as we pan the camera through it and hold on to your papers. Because it also understands humor. Yes, that's right. This needs careful prompting, but it can do it. Here we are looking at an image where people are getting amused by Obama's little prank with the scale. With a little guidance, it successfully identifies what is unusual and amusing about this image. But we haven't broken it yet. And now you will see that not even this technique is perfect. Let's try to mess with this little AI and break its brain. How do we do that? Well, of course with silly, irrelevant questions. We can even ask it what it can see outside the window and it says a parking lot. Well, not quite. And whom is this person texting? It says of course the driver. Well, you cannot possibly know that little AI. And once again, this is an incredible paper. Perhaps a step towards the highly coveted general intelligence that so many people think we will never reach. I am stunned. So what do you think? Does this get your mind going? Let me know in the comments below.
What you see here is a report of this exact paper we have talked about which was made by Weights and Biases. I put a link to it in the description. Make sure to have a look.
I think it helps you understand this paper this better.
Weights and Biases provides tools to track your experiments in your deep learning projects. Using their system you can create beautiful reports like this one to explain your findings to your colleagues better. It is used by many prestigious labs including OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get.
A free demo today.
Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you.
Thanks for watching and for your generous support and I'll see you next time.
