Researchers at the University of Tübingen found that neural networks recognize objects by texture while humans use shape. Training on both normal and texture-swapped images made ResNet50 classify more like humans and actually improved accuracy.
Scientists at the University of Tübingen.
ResNet50.
Stylized ImageNet (created by applying style transfer to ImageNet images).
$3 per month.
Dear fellow scholars, this is two minute papers with as convolutional neural network based image classifiers are able to correctly identify objects in images and are getting more and more pervasive. Scientists at the University of Tubingen decided to embark on a project to learn more about the inner workings of these networks. Their key question was whether they really work similarly to humans or not. Now, one way of doing this is visualizing the inner workings of the neural network. This is a research field on its own. I try to report on it to you every now and then and we talked about some damn good papers on this. With more to come, a different way would be to disregard the inner workings of the neural network. In other words, to treat it like a black box, at least temporarily. But what does this mean exactly? Lets have a look at an example. And in this example our test subject shall be none other than this cat. Here we have a bunch of neural networks that have been trained on the classical ImageNet dataset and a set of humans. This cat is successfully identified by all classical neural network architectures and most humans. Now onwards to a grayscale version of the same cat. The neural networks are still quite confident that this is a cat. Some humans faltered. But still nothing too crazy going on here. Now let's look at the silhouette of the cat. Whoa. Suddenly humans are doing much better at identifying the cat than neural networks. This is even more so true when we are only given the edges of the image. However, when looking at a heavily zoomed in image of the texture of an Indian elephant, neural networks are very confident with their correct guess where some humans falter. Ha. We have a lead here. It may be that as opposed to humans, neural networks think more in terms of textures than shapes. Let's test that hypothesis. Step number one. Indian elephant. This is correctly identified. Now cat. Again, correctly identified. And now hold on to your papers. A cat with an elephant texture. And there we go. A cat with an elephant texture is still a cat to us humans, but is an elephant to convolutional neural networks. After looking some more at the problem, they found that the most common convolutional neural network architectures that were trained on the ImageNet dataset vastly overvalue textures over shapes that is fundamentally different to how we humans think. So can we try to remedy this problem? Is this even a problem at all? Neural networks need not to think like humans. But who knows, it's research. We might find something useful along the way. So how could we create a data set that would teach a neural network a better understanding of shapes? Well, that's a great question, and one possible answer is so style transfer. Let me explain. Style transfer is the process of fusing together two images where the content of one image and the style of the other is taken. So now let's take the ImageNet dataset and run style transfer on each of these images. This is useful because it repaints the textures, but the shapes are mostly left intact. The authors call it the stylized ImageNet dataset and have made it publicly available for everyone. This new dataset will no doubt coerce the neural network to build a better understanding of shapes which will bring it closer to human thinking. We don't know if that is a good thing yet, so let's look at the results. And here comes the surprise when training a neural network architecture by the name ResNet50 jointly on the regular and stylized ImageNet dataset. After a little fine tuning, they have found two remarkable things. One, the resulting neural network now sees more similarly to humans. The old blue squares on the right mean that the old thinking is texture based. But the new neural networks denoted with the orange squares are now much closer to the shape based thinking of humans, which is indicated with the red circles. And now hold on to your papers because two, the new neural network also outperforms the old ones in terms of accuracy. Dear fellow scholars, this is research at its finest. The authors explored an interesting idea and look where they ended up. Amazing. If you enjoyed this episode and you feel that a bunch of these videos a month are worth $3, please consider supporting us on Patreon. This helps us get more independent and create better videos for you. You can find us at patreon.com twominutepapers or just click the link in the video description. Thanks for watching and for your generous support and I'll see you next time.
