An AI agent that literally does nothing can beat game opponents 86% of the time by triggering off-distribution activations in their neural networks. Dr. Károly Zsolnai-Fehér explains how adversarial attacks exploit the blind spots in machine-learned game players.
86% (compared to the normal red opponent's 47%).
May 9, 2020.
Dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai today it is almost taken for granted that neural network based learning algorithms are capable of identifying objects in images or even write full coherent sentences about them. But fewer people know that. There is also parallel research on trying to break these systems. For instance, some of these image detectors can be fooled by adding a little noise to the image. And in some specialized cases, we can even perform something that is called the one pixel attack. Let's have a look at some examples. Changing just this one pixel can make a classifier think that this ship is a car or that this horse is a frog, and amusingly, be quite confident about its guess. Note that the choice of this pixel and the color is by no means random and and it needs solving a mathematical optimization problem to find out exactly how to perform this. Trying to build better image detectors while other researchers are trying to break them is not the only arms race we are experiencing in machine learning research. For instance, a few years ago DeepMind introduced an incredible learning algorithm that looked at the screen much like a human would, but was able to reach superhuman levels in playing a few Atari games. It was a spectacular milestone in AI research. They also just have published a follow up paper on this that we'll cover very soon, so make sure to subscribe and hit the bell icon to not miss it when it appears in the near future. Interestingly, while these learning algorithms are being improved at a staggering pace, there is a parallel subfield where researchers endeavor to break these learning systems by slightly changing the information they are presented with. Lets have a look at OpenAI's example. Their first method adds a tiny bit of noise to a large portion of the video input where the difference is barely perceptible. But it forces the learning algorithm to choose a different action than it would have chosen otherwise. In the other one, a different modification was used that has a smaller footprint but is more visible. For instance, in Pong, adding a tiny fake ball to the game can coerce the learner into going down when it was originally planning to go up. It is important to emphasize that the researchers did not do this by hand. The algorithm itself is able to pick up game specific knowledge by itself and find out how to fool the other AI using it. Both attacks perform remarkably well. However, it is not always true that we can just change these images or the playing environment to our desire to fool these algorithms. So with this, an even more interesting question, is it possible to just enter the game as a player and perform interesting stunts that can reliably win against these AIs. And with this, we have arrived to the subject of today's paper. This is the youe Shall Not Pass game, where the red agent is trying to hold back the blue character and not let it cross the line. Here you see two regular AIs duking it out. Sometimes the red wins, sometimes the blue is able to get through. Nothing too crazy here. This is the reference case, which is somewhat well balanced. And now hold onto your papers, because this adversarial agent that this new paper proposes does this. You may think this was some kind of glitch that and I put the incorrect footage here by accident. No, this is not an error. You can believe your eyes. It basically collapses and does absolutely nothing. This can't be a useful strategy, can it? Well, look at that. It still wins the majority of the time. This is very confusing. How can that be? Let's have a closer look. This red agent is normally a somewhat competent player. And as you can see here, it can punch the blue victim and make it fall. We now replaced this red player with the adversarial agent, which collapsed. And it almost feels like it hypnotized the blue agent to also fall. And now squeeze your papers. Because the normal red opponent's win rate was 47% and this collapsing chap wins 86% of the time, it not only wins, but it wins much, much more reliably than a competent AI. What is this wizardry? The answer is that the adversary induces off distribution activations. To understand what that exactly means, let's have a look at this chart. This tells us how likely it is that the actions of the AI against different opponents are normal. As you see, when this agent named Zu plays against itself, the bars are in the positive region, meaning that normal things are happening, things go as expected. However, that's not the case for the blue lines, which are the actions when we play against this adversarial agent, in which case the blue victim's actions are not normal in the slightest. So the adversarial agent is really doing nothing. But it is doing nothing in a way that reprograms its opponent to make mistakes and behave close to a completely randomly acting agent. This paper is absolute insanity. I love it. And if you look here, you see that the more the blue curve improves, the better this scheme works for a given game. For instance, it is doing real good on kick and defend, fairly good on Sumo humans, and that there is something about the Sumo Ants game that prevents this interesting kind of hypnosis from happening. I'd love to see a follow up paper that can pull this off a little more reliably. What a time to be alive. What you see here is an instrumentation of this exact paper we have talked about which was made by Weights and Biases. I think organizing these experiments really showcases the usability of their system. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that if you're an academic or have an open source project, you can use their tools for free. It really is as good as it gets. Make sure to Visit them through wnb.com papers or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for their long standing support and for helping us make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
