EP84 How to Secure Artificial Intelligence (AI): Threats, Approaches, Lessons So Far

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at GCP, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. Please rate our podcast if you think highly of it. And if you don't think highly of it, let us know quietly, not through the ratings. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is episode 3 about AI that we've done so far now? And I think this one is quite a fun one because we get to get into the weeds of some actual attacks on AI. We get a framework for thinking about AI, and all the people with an AppSec background get to feel right at home.

Right. But also we had one moment when Tim was scared and one moment when Anton was scared, right?

And neither had to do with seeing a snake.

The Anton moment was connected to a snake, but Tim's moment was about a car, right?

That's right. I was worried about getting run over.

I think we are spoiling too much, but—

No, I think listeners are going to think we're insane until they listen to the rest of the episode. Perhaps we shouldn't have any further ado. Please welcome today's guest. We are joined today by Alex Polyakov, the CEO of Adversa, whose website is adversa.ai. Alex, I'm really excited about this research you put together by analyzing some absurd number of papers on AI attacks over the past decade. Give us the rundown. What did you learn?

Yeah, I think so. I'll start with a broader answer and then jump into the details. I see like 3 waves of research in secure AI. So kind of before 2015, when we found like the risks and vulnerability categories, and there were some examples of each category like poisoning attacks, evasion attacks, instruction attacks. And it reminds me like application security days when we found like there are injections, out bypasses, and et cetera. And the next wave, it's kind of 2015 till 2020. So, we might call it the era of vulnerabilities and variation of attacks in each category. Like, we found that there are targeted evasion attacks or untargeted evasion attacks. So, think of it like we have different types of injection in AppSec era, like SQL injection, buffer overflows, and so on. And what started, like, after 2020, it's kind of the era of the attacks and exploits when we saw the particular methods and different attack goals. So think of it like we found the file read attack via double-blind SQL injection in Oracle database. So this is kind of like how it's evolved. So our report covered basically the first 2 waves. So briefly, what we found is that almost every type of AI was demonstrated to be vulnerable. So various tasks like classification, regression, reinforcement learning, It's also different types of algorithms like logistic regression and neural networks and various types of data like images, audios, and et cetera. Second thing that we counted, there were like at least 10 different attack categories among 2,000 research papers. And the last is that there were no one-size-fits-all protections. So we don't have any idea how to solve it.

So wait, this sounds just like application security. You're describing an evolutionary journey. You're describing attacks against different kinds of applications. Applications, e-commerce, hosting, whatever. You're describing attacks on different kinds of backing languages as if we had just discovered that, yes, PHP and Python-based web apps are both vulnerable. This is great. So what's old is new again.

Yep. 1998, just replace applications with AI or something, right? I don't know.

Yeah, yeah. It really reminds me the same.

So hopefully you have a link to some kind of summary study at the end. So we'll appreciate getting it for our resources after the podcast. When I hear people analyzing attacks or trying to invent AI safeguards, especially in academic papers, I'm always nervous. Like, what is their threat model? What are they thinking about? So in some recent podcasts we've done on AI security, some people obsessed about supply chain for data or other things. So how do you approach thinking about the threats, threat modeling, threat assessment? Which threats matter? which threats may matter in the future? Could you start us out here?

Yeah, I was actually like in the previous podcast with Nicholas, you mentioned like the important thing that AI systems will change and the threat may differ in 5 to 10 years. And I kind of agree that techniques can change, but at the high level, what I think is that AI weaknesses in general will be more like human weaknesses rather than the software weaknesses. Like the problems of the human brain because AI systems will be more and more like a human brain. We already see examples of it like the adversarial perturbations. They look really like hallucinations or psychedelic effects. There were even research papers demonstrating that you can fool the human perception with a method similar to evasion attacks. If you have people who have arachnophobia, And I've tested it with such people. So, you change the picture in such a way that it's snake, it looks like snake, but it's adversarially changed to spider. So, the people who have arachnophobia, they don't understand there's a spider, but they say that it's something wrong with this picture and they can feel it emotionally. So, going back to the original question, like a threat model, I would say that we should start with the question, why? What is the goal of the attack. There are 3 high-level things. You can manipulate decision. You can fool the system the same as they can fool the human. You can retrain the behavior, like poison the system, or you can guess some secrets, like extraction attack. This is the high level. The next question you have is what exactly you're targeting. It can be the model itself. It can be the data. It can be the algorithm, like how the model really works. And then the next thing is how exactly you're going to exploit this vulnerability because there are like 100 methods to perform even simple evasion attacks. And they differ because the different limitations of the model. For example, if you have a digital attack, like attacking the digital world against the computer vision classification, you can change every pixel but a little bit. But if you have a physical attack, you don't have control against each pixel. So you just can wear the mask. So you only can change the pixels in the mask, but you can change them in any way. So this is just one example of how attacks differ in the threat model.

So when you started to say that attacks on AI are kind of like attacks on humans, I was imagining a world where we started phishing the AIs, but it doesn't sound like that's quite the case. It sounds more like the ways that AI perception can be fooled are similar to the ways that human perception can be fooled.

Yes, visually.

But in terms of the stealing attacks, they look similarly to how people guess if you're lying or not. It's the same part. Yeah. So you ask different questions about the topic, but you ask the question differently. And by the responses, you can understand if people are lying or not. And the information-stealing attack from AI model looks kind of the same. You're sending a lot of similar requests, and this is how you understand the result. So there are a lot of similarities here.

That's quite interesting.

Interesting. And I think that for some reason, when you describe that the visual attack against the AI kind of sort of confuses and affects humans, I'm getting like horrible cyberpunk memories of the 1990s books, you know, Neal Stephenson and whatnot. Like, I'm having some freaky flashbacks to mind viruses and all that. So I don't know, it's kind of cool, but it's cool in a strange surreal way.

Wow.

But we might get there sooner or later.

We probably would get there sooner rather than later. That's the other freaky part, right?

Yeah, that's true. Yeah.

Okay. So let's keep talking about these cyber-physical attacks on computer vision because I think that's an interesting area, but focusing on computer vision, not human vision. Is this real? I remember seeing some research about this a couple of years ago, people putting stickers on stop signs to confuse cars or whatever. Is that a real thing to worry about? Is that happening? Can I put on face paint and become invisible to the cameras? Where are we at?

Yeah, that's definitely a great question. And this is one of the questions we wanted to learn since the beginning of Adversa because our idea was to understand how realistic are those academic papers and make the bridge between security-related threats and academic vulnerabilities. And yes, it's possible. And probably the question now is for attackers to find how— I call it exploit market fit. So how exactly they can exploit the vulnerability. And if it's a facial recognition system, it's obvious you can fool the system by having a mask that can hide you from surveillance cameras. You can create special glasses that can hide you or they can basically impersonate you as a different person. In military, there were examples of hiding planes and tanks from AI drones. It's also obvious why it's critical. In terms of autonomous driving systems, there are also examples like you can put a sticker on a car And you can crash into this car. So those are the real scenarios. But now the technical—

Real, wait, wait, real meaning likely. Real meaning— real doesn't mean that they happened already, right? Real means they can happen.

Some of those already happened, like with the face recognition systems. And there were examples in China. So the real ones.

But nobody's putting a sticker on the back of my car to make it get run over by a self-driving semi, right? Okay, I'm gonna check my car for stickers from now on. I've got a new kind of paranoia today. Thank you, Alex and Anton. Really appreciate it, guys.

Yeah, nobody's doing it now, but you know, like the taxi drivers, they will be against autonomous cars. I can guarantee, like especially in the places like Indonesia or somewhere else, like where it's the the very important part of income for people. And if you gave them this ticket and say, hey, like, this is your way to overcome those big companies and enterprises, like, they can do that. You know.

What I love about that is that it has a good economic motivation behind it, like all good hacking stories do. So that's great.

Yeah. Yeah. But now, like, the technical part, like, how easy is that? Because if you simply take the image and create this attack in the digital world and then print it and try to apply it, it won't work because there are— when you print, the colors can be different. The cameras can have different color rendering. You have distance, angles, everything. You need to take into account all those things. When you create the exploit, you need to train this exploit to overcome all those problems. But it's possible. It's just a more complex way of exploiting things.

But it's also important that visually you're not just printing a face. You may be printing a weird picture that does not confuse the human in this case but does confuse the AI algorithms, right? That's what roughly you're describing, right?

Yeah, yeah. You just print glasses or earrings or something. And it's a fun thing that Exploit now can be everything. So the t-shirt can be exploited, the mask can be exploited, like the stone can be exploited. And this is fun.

Yeah, this is very fun. This is like a— this— but it does give me some kind of cyberpunk book from the '90s kind of vibe, which is interesting. So it's gonna probably be dealing— we'll be dealing with that in the 2030s, even though the science fiction writers wrote about it in the '90s. So I want to turn us to a bit of a— maybe not boring, nothing about this is boring, but kind of a more traditional subject. So sometimes we hear that AI system today is just like a big, messy enterprise application that does data analysis. And they kind of equate AI/ML processing pipeline with like a big, messy enterprise app. So when you think about it and when you do your research, What are the differences between protecting just some other big enterprise application versus an AI application? Is there any way to think about it?

Yeah, sure. So there are a lot of similarities. And most of the things I mentioned now, it's about the algorithms itself. And of course, AI consists of the preprocessing, like the servers, APIs, and et cetera. Those are also the similar things. Also, the similarities are because we have the same approaches we can use, like predict, prevent, detect, respond from Gartner. It definitely works. And it's necessary because unfortunately, we don't have one-size-fits-all protection. So we need to have like multiple approaches to defend. And but one thing, you know, that was mentioned already by Carlini is that by making AI more secure, you're actually making it more robust sometimes.

Yeah.

And this is why it's important to protect AI at the beginning during the development stage and not in the detection. What's also good is that you can actually quantify how you protect the system. So, you can specify a threat model, environment, size of perturbations. And you can have almost like mathematically proven defense at some point. So what I really like here in AI space, comparing to the traditional cybersecurity space, is that you have a firewall and vendors saying that, hey, this— you can put this box and it can protect you from everything, but you have no guarantees like at all.

Yes.

Here you have a chance to have some kind of provable guarantee against particular types of attack mathematically. I'm not saying it's easier. It's a hard thing, but it kind of looks like it's possible comparing to what we have.

But it also requires a degree of forward thinking. And of course, Google is expected to have forward thinking, but perhaps many other AI designers would not necessarily have this type of forethought to do it step one, right? Because ultimately, if you're saying some of the controls, some of the safeguards has to be designed in, Day 1, day 0, or whatever, and not later when you have the system in operation, right? That has been difficult for us in the industry, right?

I think about the development part, like the code scanning practices. Yeah. So like the vulnerability, like finding the vulnerabilities and trying to protect from those vulnerabilities, like something like this. So, but not like, you know, the detect and respond. From day one.

Yes.

So on this topic of securing AI, whose job is it? How should organizations approach the problem of allocating who's in charge of making the AI secure?

That's a great question. And I would love, I personally would love to hear the different approaches, like how companies are thinking about that. But what I hope how it will be is that the product security people will be responsible for that. Because we can copy so much from product security and we can teach product security people to understand more about AI security because basically AI is like a new way of developing things. It's also called sometimes Software 2.0, and every development will have some sort of ML practices and like code will be the commodity and will be more and more ML and code will be commodity. But the approaches that we have in product security are very good and we can use them. But it's also important to have support from CISOs because they won't be like responsible of it a lot unless it's an AI-driven company, you know. But product security people should have support from CISOs and from AI developers. And I think that I see a lot of interest from AI developers about security, like rather than the security guys. And that's great because it's the first time.

That's a great sign. Yeah.

Yeah. It's the first time in my life I see like This and not vice versa.

That's awesome to see. Is secure AI the same thing as trustworthy AI? And how do we build trustworthy AI in addition to secure AI?

I would say that it's a part of trustworthy AI because trustworthy AI, like at the high level, it consists of probably like the explainability type of thing and reliability, explainability. Like the second thing is like security, privacy, and safety. And the third is ethical bias, responsible, etc. So at the high level, it's a part of it.

Yeah, that question came to mind because I was listening to a podcast between Andy Ellis and Allison Miller last night where she talked about the role of trust in her job definition. So it's interesting to pick at these boundaries in the context of AI.

Right. But I think that it's a bigger— I don't think that in the next 10 minutes we'll resolve the trustworthy AI here.

No, I don't think we will.

Frankly, I'm pretty far from being an expert in this.

I think we should move along.

Yes, I think it's a good idea. So one thing to touch is when we, in the other podcast episodes, we've heard from people who, of course, pointed out that some of the components of AI may be kind of opaque. Sure, there's AI transparency and other stuff, but when we hear of some systems that have opaque, non-transparent components, there's a question of vulnerabilities, right? How do we disclose vulnerabilities? How do we discover vulnerabilities? So given that we touched on AI being a big complex application, Alex, how do you envision vulnerability research and vulnerability disclosure? Is it only about open source AI or only about APIs? Like, how do we go about it?

Is somebody building the AI Nessus?

Well, yeah, you just have to bring the 1990s back into the conversation, right?

Can't help it.

Yeah. Oh yeah, that's actually a very important question I wanted to bring more awareness into, but Because, like, for example, if I found the vulnerability in open API of some cloud provider, so will there be the CVE for that?

Mm-hmm.

And if yes, will it be the CVE for this API or for the vulnerable model? And who should I contact? Because I saw the papers with examples of real APIs, and I saw from the responsible disclosure perspective from the cybersecurity space, like, it's kind of not ethical. And if we have those research papers in arXiv, but we don't have the CVEs for those vulnerabilities. And why is it really important? Because I didn't mention one very important property of attacks on AI. It's called transferability. So if you have exploit for one model, there is a big chance that it will work for other models. It will have less effect probably, but you can use the ensemble of models. You can train your exploit in an ensemble of models, and the more models you have in your ensemble, the more chances that your exploit will work. So, the problem of responsible disclosure is really important, and I hope that sooner or later we will have all those CVEs for Yeah, that's really interesting that attacks are transferable.

There's actually really nice research out of Project Zero about similarity on patching related to zero days and how one zero day, when patched, might still have other zero days related to it. So this is not too surprising to me that one attack on AI would be transferable to other generations of the model. That's fascinating.

So that means that you do envision some form of vulnerability disclosure for AI in the future, right? Like, it may happen. There are venues and mechanisms for doing it, right? Like, it's possible.

Yeah, it's definitely possible, but it should be some more efforts into this area. And like, one company cannot solve this. It's an industry thing.

That makes sense. It's kind of maybe in the standards or anything. So while we're discussing this, I want to have some advice kind of for the defensive side perhaps and touch on constructing or building an AI security program. So what should people do first when they're embarking on an AI security program? Well, who should have such a program? And just give us a— I mean, given the shortage of time, just give us a broad outline. What would the program look like?

Sure. So I think that the first step is basically to find a responsible person or the committee in the company to deal with that. The second will be to start with the threat model and understand, like, what are the threats exactly. And you may use this, our approach, it's not our approach, like, but why, what, and how, like a cheat sheet to understand what are the threats for your system. And then basically try to, like, run some pen tests and to understand what are the real threats and what are not real threats. And you can, at least you can be aware of the risks and you can quantify them. Like, those are the real ones, those are not real ones, and so on. So this is probably like the first simple steps.

Yeah. And as far as, do you think every company that uses AI or AI techniques or AI methods or is building AI needs to have it? Who should do it? Like, if somebody just signs up for some online AI application or AI platform, does it mean they have to have an AI security program?

So we call it like mission-critical AI. If AI systems for you are solving some kind of mission-critical task, and it doesn't matter if you develop AI system, if you sell AI system, or you consume AI system. So this is where we put the separation, like mission-critical AI application can be content moderation, face recognition, something related to people's lives and critical decisions.

Okay. No, that makes sense. I mean, basically, especially if you are— but if you are designing your own AI, then almost certainly you would have to have a program. At least that's my kind of a gut feel or impression that if you're not just using the off-the-shelf models from somewhere.

Yeah, 100%.

Yeah.

Perfect.

So our typical closing questions. Give us one piece of advice for, well, in this case, related to AI security, And of course, give us some links to further reading. We promise to put the resources up there with an episode.

Sure. So one piece of advice, I think, in addition to all the first steps I've mentioned, I think that we can use examples like adversarial examples to basically retrain the system. And sometimes it helps to make the system more protected and even more Robust. And in terms of the links and etc., so we obviously have the white paper that we mentioned. We have a blog where we post weekly news about secure and trusted AI and monthly review of research papers. You don't need to read all this archive craziness, so you can just read the most important ones. Also, I would like to mention that AI/AAIC repository. So it's a database of different threats in AI. It's very good just to see what's happening and what are the real threats. And I think last one, we just recently, unfortunately, we were not able to touch it in the details, but we recently launched the competition against AI systems, like facial recognition systems in the MLSec. So you can join this competition and even win the prizes. And it's a really fun thing.

Perfect. We'll definitely put a link after the episode when we launch it. Alex, thank you very much. This was moderately scary, but very interesting. So definitely appreciate you coming.

Yeah, thanks for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Bye.

Security.

Security.