A CNN extracts visual features from an image, then an RNN generates words describing it. The model reaches 90% accuracy on captions, about 4% behind humans. See successful examples like "man throwing a ball" and entertaining failures.
Around 80%.
About 90%, which is roughly 4% off human performance.
Dear fellow scholars, this is two minute papers with neural networks can be used to learn a variety of things. For instance, to classify images, which means that we'd like to find out what breed the dog is that we see on the image. This work uses a combination of two. A neural network variant that is more adapted to the visual mechanisms of humans and is therefore very suitable for processing and classifying images. This variant we call a convolutional neural network. Here's a great web application where you can interactively train your own network and see how it improves at recognizing different things. This is a dataset where the algorithm tries to guess which class these smudgy images are from. If trained for long enough, it can achieve a classification accuracy of around 80%. The current state of the art in research is about 90%, which is just 4% off of humans, which is who have performed the same classification. This is already insanity. We could be done right here, but let's put this on steroids. As you remember from an earlier episode, sentences are not one thing, but they are a sequence, a sequence of words. Therefore, they can be created by recurrent neural networks. Now, I hope you see where this is going. We have images as an input and sentences as an output. This means that we have an algorithm that is able to look at any image and summarize what is being seen on the image. Buckle up because you're going to see some wicked results. It can not only recognize the construction worker, it knows that he's in a safety vest and is currently working on the road. It can also recognize that a man is in the act of throwing a ball. A black and white dog jumps over a bar. It is not at all trivial for an algorithm to know what over and under means because it is only looking at a 2D image. That is the representation of the 3D world around us. And there are of course, hilarious failure cases. Well, a baseball bat. Well, close enough. There is a very entertaining web demo with the algorithm and all kinds of goodies that are linked in the description box. Check them out. The bottom line is that what we thought was science fiction five years ago is now reality in machine learning research. And based on how fast this field is advancing, we know that we're still only scratching the surface. Thanks for watching and I'll see you next time. Oh, and before you go, you can now be a part of two Minute Papers and support the series on Patreon. A video with more details is coming soon. Until then, just click on the link on the screen if you're interested. Thank you.
