EP68 How We Attack AI? Learn More at Our RSA Panel!

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of our cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday afternoon Pacific time, please hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is our RSA episode, and I guess after over a year of podcasting now, we were able to project plan ahead well enough to release our RSA episode the week of RSA. That feels like a podcasting win. Last year we were way off.

Yes, for sure. I think we released our RSA episode in like November or something.

That was not good. So this one, listeners, this is a fun episode because we're releasing it the Monday the week of RSA, and we're interviewing a guest who goes on to speak at RSA later this week, assuming everything goes to plan and nothing crazy happens in the universe. And the topic I found Fascinating. This is our second, third, fourth— we've talked about AI a few times now on the show, but this time it's kind of a technical conversation. What do we get into today? What's the topic area other than just AI and RSA?

I would say, and I'm going to jump a little bit ahead of the discussion, but there was one particular bit that was quite mind-blowing in the discussion. And I'm going to try not to give too much of a context, but our guest described a measure that increased security and coincidentally increased resilience? Because a few weeks ago, I've heard of many examples and people say, well, if you encrypt this, then something becomes more fragile from an operational point of view. And people are, at least in our industry, are used to seeing 2 things. Either they see security improves, resiliency decreases, or they see resiliency improves, security coincidentally improves. But I've never seen an example where security improves intentionally, and then resilience improves coincidentally. Of course, we are talking AI models, so it's probably based on alien science anyway. But this, to me, was the most interesting bit.

This episode, like the confidential compute episodes, features a lot of alien science. So with that teaser of alien science and with resiliency, let's welcome today's guest. Today, we're joined by Nicholas Carlini, a research scientist here at Google. When I hear research scientist at Google, that's pretty cool because there aren't a lot of those, it turns out. And what we're talking about today is securing AI systems. So Nicholas, what's a threat model for a large-scale AI system? How do we think about the needs? What are the attacks? What's going on there?

Great to be here. I think the main question of a threat model in AI is trying to come up with, just like any other threat model in security, what is the likely kind of attacks that are going to be possible and what do we have to be worried about? And usually this is relatively similar to any other security threat model, but you have to now be worried about new kinds of threats that might not have been possible in normal security settings.

So what's a new kind of threat that wasn't possible before?

One of the big kinds of threats you might have to be worried about is in machine learning, you usually have to train on datasets that might not be entirely under your control. And what this means is that an adversary might be able to modify the data that goes into your algorithm So that your algorithm is now going to produce some incorrect errors. This is called a poisoning attack because of the data that some malicious party fed the system. Usually when we write normal code, some person sits down in front of a computer and writes the algorithm down from top to bottom, and we assume this person knows what they're doing. But in the case of poisoning, some person writes down the training algorithm, but then it takes in a lot of other data from who knows where, and we have to hopefully rely on the fact that this data was good. And is not going to try and make the system behave incorrectly.

But presumably there's a lot of unintentional mistakes and just sort of non-intentional badness that also comes from bad training data, right? So how do you tell AI that learned something bad we want it not to learn, like bias, versus actual attack where somebody is purposefully trying to have it learn something bad? Am I making sense or am I rambling?

No, this makes a lot of sense. There is a big difference between average case and worst case. Robustness or security. So you can think of this being— we have this in normal security as well, right? Let's suppose I write some function that behaves well in all situations, except when the length of one input is exactly 3 bytes longer than the length of some other input. The probability that this happens due to random chance might be very, very low. We might say there's no bug here in most settings. But from a security perspective, this is terrible because someone can arrange for this to happen every single time. and can then cause it to exploit the system. In machine learning, we have the same thing. You can have systems which are very resilient to noise in the labels, so that data might just be incorrectly labeled randomly, and it can behave perfectly fine. Actually, it turns out there's this amazing property that machine learning models are robust to something like 50% label noise. Half of the data can be labeled incorrectly, and you can still learn models that are roughly as good as—

I did not know that.

Yeah, this is a very nice paper.

It got best paper.

I didn't know it was that high.

Actually, that's kind of cool.

Yeah, it got best paper at one of the top machine learning conferences. a couple of years ago that you could take some dataset, inject a huge amount of extra data, label it all wrong. As long as you label it wrong randomly, the model just becomes slightly less confident on all of the data, but it's not any worse. But as soon as the adversary can control even a very small fraction of a dataset, in a worst-case setting, they'll sort of set things up in just the wrong way so that you can cause various attacks. And this is true not only in the data poisoning setting, but in other settings as well. So machine learning models, if you give them an input plus some small amount of noise, they're very robust. If you add a small amount of noise to an input, it will behave just as good as the normal input. But you can add very, very small crafted noise called adversarial perturbations so that you can have images where you only flip the lowest bit of each pixel, but change the label to any label that you want. So you can take an image of a dog, flip the lowest bit of each pixel, and make a classifier that's a perfect classifier in any normal setting classify this as a cat.

Wow.

But only because this is sort of adversarial in this one setting. It's very, very good in the randomized setting. It's very, very bad in the worst-case sort of setting here.

So this is an example of something that's legitimately unique to AI, which is— so Tim, I'm having an epiphany. And sorry, I'm talking over our guest here.

You have your epiphany.

I'm having my epiphany because remember the question that I used to torture our previous AI guest? I said, What's the difference between attacking AI and attacking a big complex data processing system? That's the difference. You don't have that type of an attack, that type of threat in your model for just a big data processing system, a CRM system that's very messy and complex. You don't have that, but in AI you do, right?

Part of this comes down to the fact that in machine learning systems, most systems work with high-dimensional data, like data that has lots of different variables. With big giant systems, normally this is hard to deal with. What's the best— if I have some big giant file, what's the best thing to do to evade some standard malware detector? I don't know. Machine learning systems are a little bit unique in that you can query them and ask specifically, you can formally ask for the gradient, which what this means is it will tell you if you ask it exactly which input should I change to make the classifier give me the wrong answer. And this is necessary in order to train these models. The way that you train a model is you say, which parameters should I update to make the model behave better? You can ask the question in reverse and ask, which parameters of the input should I change to make the model behave as poorly as possible?

Of course.

And so the exact same process that you use to train these models, that's necessary to train them, is the same mechanism that you use to attack them. It's because you can say, I have some big giant image. My image is a million dimensional. It's some megabyte of data. But I can say, exactly which pixels do I change to make the model behave incorrectly? And it will tell you. Flip these pixels and it will give you the wrong answer. And this is great from an attacker perspective, but very, very annoying from a defender perspective.

That's fascinating.

And even from the detection point of view, like observing that it happened, it's almost like perfect knowledge in one direction, but really obscured knowledge in the other direction. Detecting that this is going on is kind of extremely hard. I'll use the polite, mild and polite term, extremely hard.

Yes. No, and people have tried to write papers that say, well, let's try and use machine learning to detect these kinds of attacks.

Right, of course.

The problem is these detectors are machine learning algorithms.

Of course.

And so then what you do is you simultaneously attack the original system and evade the detector.

Right.

And it's generally not so much harder than just attacking the machine learning system straight out of the box.

Oh no, that's terrible.

Yes.

But it's exciting. Come on, dude. Are you a security person or not, Tim?

Terrible?

It's exciting.

No, it's terrible in that, oh my God, it sounds hard to defend. And Anton, last time I checked, I was still on the blue team, not the red team.

Well, yeah, you're right. You're right. But it's like a red team excitement a little bit.

Yes. So, okay. So we're getting all over the place right now, which is always fun. Anton and I are clearly very excited about this, but I want to reel us back in a little bit and make 2 things concrete. One, we're doing this episode because RSA is coming up and you're speaking at RSA, and I would be remiss not to mention that. People should see the talk at RSA. But 2, I want to make this concrete by saying, Okay, this is all very theoretical. How do we know that we need to mitigate these things? When we're doing threat modeling, one of the critical things is we have realistic threats in mind. How do we know if these are realistic?

Yeah, this is a very good question. Right now, I don't think I know. I think one of the big things that we're trying to do with research is to answer the question, what threats are practical versus what threats aren't? And this changes over time as the capabilities of systems change. We try and stay a little bit ahead and guess what potential use cases might be, and then as a result, guess what threats might become possible. So this poisoning threat model I was mentioning at the beginning, where some adversary might control a dataset, this work has been going on for a while, but I didn't really believe in the threat model for a long time because usually we were training on small datasets that were collected by researchers who knew what they were doing and everything was fine. But recently within machine learning, there's been a trend to train on bigger and bigger datasets where you require scraping the internet. And now all of a sudden, poisoning is a big problem because if you're scraping the internet, someone on the internet is probably malicious.

That's a good point.

Bad assumptions. And can do bad things. And so what is a threat at any point in time changes constantly as the systems that we have get more and more powerful. And I don't necessarily know what all of the threats are that we should worry about, but this is one of the main things that we try and think about. What I really like to do is to think about what potential attacks might be possible and try and show how bad it could be, so that even if we don't have a system that relies on it right now, we can make sure we don't build a system that would rely on something that ends up being not true.

Wait a second. So there's a bit of a fascinating timing angle to that because you may have a threat that's absolutely in the theoretical bucket, but there's a very clear path that will be treaded by people in 5 years. So it's almost necessarily become real in 5 years. That's a scary thought. Oh, okay. Exciting thought.

Yeah, definitely. This is what part of what I like about research is. In some part, what you're trying to do is predict how people are going to be using these tools and then try and predict, well, suppose that they actually do use these tools in this way, then what kind of attacks will be possible in the future so that we can hopefully first understand how bad it is. Maybe it turns out it's just not very bad. Maybe this attack doesn't work and you don't have to worry about it. But if it turns out it is bad, then we would like to know how do we defend against it before people actually rely on it so that we're not caught off guard with thousands of people relying on a tool that turns out, oops, it's vulnerable. We have to go sort of rebuild everything from the ground up.

But can you share a little bit of a methodology about how you would assess something in terms of becoming a real threat in 5 years? Would you look at some kind of extrapolated charts, curves curving up, and then suddenly they intersect with what's really— sorry, I'm making this up a little bit.

Yeah, that's fine.

So how would you approach it?

No, this is exactly what we try and do. I mean, I don't think that we necessarily look at these things as scientifically as we might like to want to. We just try and get a general sense of where does it look like progress is being made most quickly. And then we try and understand, assume that this trend continues for the next couple of years, what will things look like in the future? And so, for example, one of the trends that we've seen over the last 5 years is the size of machine learning models has been doubling every couple months. Far faster than Moore's Law. And so there's a question of, this has been happening every year since 2015, every couple of months the size of the biggest machine learning model doubles. It doesn't look like it's going to stop anytime soon. This sort of begs the obvious question, well, what are the security implications of this? And we try and spend some time thinking about that. Same simple thing with training data doubles every, maybe every year or something like this. And we try and look at what the current trends are and predict how this is going to impact things in the future.

Okay, so you're trying to predict the future?

Yes.

I've heard that's hard.

Yes, it's very hard. Sometimes you get it wrong. You think, here's something that will be really, really important and impactful, and it turns out no one uses the system. Then the security of something no one uses doesn't really matter because no one uses it.

That's fair. Rather than predict the future, let's talk about the present or the past. What are we actually seeing out there? What are the real threat-derived lessons for securing AI? Because I hear threat Informed defence is popular these days?

Yeah, no, this is a good question. And I spend most of my time really on the research side of things, so I don't think I'm the best person to be able to say what's actually going on right now. I can tell you from the papers that have been written that most people are worried about maybe 2 or 3 kinds of threats. One of them is this poisoning one we spent a couple minutes on. One of them is this evasion one we spent a couple minutes on, where you can modify inputs to make the models be misclassified. And then the final one that people are worried about are various kinds of what's called extraction attacks, where you can imagine I spend as a company lots of money training a big giant model, and then someone else goes and they interact with my model and they use this so that they can get a copy of my model that's as good as my copy. And now they can go use this without having to pay me for my model.

And without spending all the money on electricity, training, data transfers, cloud costs.

Yes.

So I'm assuming that with some of the models you are talking about nice 7, 8-digit amounts of money to train them. But if I can steal your model by interacting with it, that'd be kind of cool, right?

Yeah, exactly.

So it's like IP theft, AI style.

Yes. So this is one kind of stealing. And the other kind of stealing you have to be worried about is let's suppose that I train a model on some sensitive dataset. Then you could imagine worrying about the training data privacy. These are the kinds of threats that we end up trying to think about from this perspective.

But that's not the very— the training data privacy to me is kind of nothing to do with AI there. It's just a sensitive dataset. We sort of know, and I'm doing a cringe on my face, we sort of know how to secure a dataset like that. There's no real magic to this one.

There's a lot of research here. Maybe we don't want to spend too much time on this, but there are things that you can be worried about here.

Makes sense. So as far as given these threats, it's kind of relatively clear to look at the threat-derived lessons. Of course, it's the privacy of training data, you limit access, you do whatever. But the whole model extraction thing, I have no idea how to even approach defending against it. This is like, do you make the model lie to some people so they don't know what it thinks? I don't know. Sorry, I'm getting carried away here.

No, you're completely right. This is a very hard question. I don't think that in the research space we know what the answer is here either. The sort of thing that you just threw out actually is a real suggestion some people have made, where occasionally what you can do is Normal people, you can sort of detect the kinds of things that they ask models. But sometimes if someone is asking for the same question, but phrased very, very, very slightly differently, hoping to probe the model to be able to learn exactly its decision boundaries, the model might be able to detect and say, this is not the kind of thing a normal person would be asking of me. I'm just going to start giving you random outputs to hope that I'm not going to disclose the little things that I've learned. And you have to be careful that you don't actually start lying to people about things that matter. I mean, it would be very bad if you had some cancer detection model that thought it was under attack and started giving incorrect diagnoses because it thought it was under attack when actually it wasn't. But this is one of the big challenges for many of these systems is how do you defend against this? And we don't have good answers right now.

But this to me is a fascinating career path for somebody who is maybe in undergrad or in grad school. Because we may well be dealing with this for the foreseeable future. I don't know, decades possibly, right? So if you're planning your career, maybe becoming an AI security scientist is a good occupation.

Yeah, it's right. It's entirely possible that it could be.

Well, it's obvious for you, Nicholas. You are there.

No, I mean, right. I mean, I think there's a couple different directions you could go with this. I think it's entirely possible that in 5, 10 years, we'll see people who are sort of the pen tester kind of directly testing machine learning models for their security. But it's also possible that in 5, 10 years, it turns out that this current type of machine learning that we're doing is not what everyone's doing. I mean, deep learning as a field only really took off 10 years ago. Who am I to say that in 10 years we're not going to have some new different kind of model that behaves wildly better than what we currently have, and people who spent a couple of years learning about this specifically might be caught behind the times? It's very hard to tell for this what it's going to look like in a couple of years.

Again, back to predicting the future, which I thought was kind of the whole point of AI was to offer predictions.

Yeah, but we're not sci-fi. Well, yeah, sci-fi, but also reality kind of. This is— these are the episodes where they sort of touch a little bit. So here's the other angle that I wanted to explore, and I think you mentioned that you'd be talking about it at RSA as well. So I feel like there's more research on how to make AI reliable, unbiased, correct, ethical, like that bucket of issues. But there's also a bucket of issues of making AI Secure, which means resistant to attacks. Are these similar, connected? Is there a big 3D Venn diagram to think about? How do we think about the approaches for secure versus reliable AI? I'd love to hear from you about this.

Yeah, so they're definitely related. So there's this very nice quote, I think it was maybe Russ Anderson or someone, where suppose that you had a system that was robust against the worst-case attacks. And so, for example, I have some self-driving car that even if there was an adversary present who could flip bits in the camera and just do arbitrarily bad things, suppose this system would not crash, then probably also it's not going to crash if there happened to be glare in the wrong way coming off of some reflection that causes the system to crash because of this. Now, so it's in some sense like the worst-case secure setting is just sort of the worst case in the randomised setting. The problem is that when people talk about randomized security, what they're often worried about is sort of the tails, but not that far into the tails. They're not worried about these once in 100 million years events. They're worried about things that happen daily or these kinds of things. So the techniques that you use to make things robust on a daily basis might be very different than the ones that you use to make them robust in the once in 100 million years randomized settings. And there's some relationship, especially in the case of machine learning. It turns out That some of the best techniques that we have to increase the security of machine learning models do this by increasing their general out-of-distribution robustness and then using—

Wait a second. Not the other way, but the security techniques boost reliability, but reliability techniques may not boost security.

So currently, one of the best techniques we have for preventing evasion makes models more robust to random noise and then uses some clever mathematical arguments to be able to introduce a technique that can provably increase the robustness to worst-case noise.

Wow.

So the way it works is you suppose I had a system which was robust to random noise and you give me an input. What I'm going to do is I'm going to add a new random noise on top of the input, whatever you gave me, and then classify that. And because my system is robust to random noise, whatever sort of perturbation you added to your input, I'm just going to mask it away with the random noise that I'm adding on top of it and make the thing work in this way. As long as it's robust to random noise, now I can sort of make it robust to worst-case noise. Of course, there's lots of details here that I'm glossing over. You have to do this sort of several thousand times and you have to ensemble it the correct kind of way. If you miss any details, then the system completely breaks. But this is one of the recent nice things that people have been doing.

I love it when research scientists start a sentence with, well, suppose. That's always going to be good when we start there. I run an AI system in production today as part of one of my security products.

Are you more afraid for it now? I'm more afraid. Are you more afraid for it now? We're getting there.

We're getting there.

We're getting there.

So I run a system in production at scale that classifies bash scripts into benign and malicious. And it works pretty well, actually. You know, we do a really good job of turning up attacks and frequently we will see attacks on popular software about a week before the CVE comes out, which is pretty cool. However, I had a conversation with one of my customer engineers this morning where he was playing with my system. And he ran some stuff and he says, how come it didn't detect this? And he kind of had that look on his face. And customers, listeners, it wasn't a happy look, right? And so I've got this explainability problem with my system of, well, that thing you're doing, that probably wasn't in our training set and it's not that scary anyway. But that's hard. So is this a general problem in AI systems? Is it hard because they're hard to explain? What's the current state of thinking on transparency in AI and how that relates?

Yeah, this is very difficult. It's one of the many big problems that people are working on, is trying to be able to develop techniques to explain machine learning models and then to understand what they're doing. And in big part, one of the reasons why it's so hard to secure these systems is because we don't even know what they're doing in the first place. Often the first step to developing a secure solution is to be able to understand exactly what the system is doing. And then once you know everything that it's doing, you can then develop a solution that fixes things.

But you fall and break your leg at the first step here, right?

Yeah, exactly. Yes. And this is why security is hard, is because we We don't know exactly what these systems are doing. And even as the more advanced we're getting in these machine learning systems, the less and less we understand what they're doing. Sort of a canonical machine learning model, you train on some labelled training dataset to solve the task you want. Like, is this malicious or is this not? What people are doing now is not even training on labelled data. They're sort of feeding just arbitrary sort of data into models and letting it label the data itself and then training on its own self-generated labels. And they've shown empirically this works and it sort of helps a lot. This self-supervised and semi-supervised learning is the most recent revolution in deep learning over the last couple of years. But now you're asking me to trust a system that not only do I not understand in the normal setting, I'm asking it to teach itself, which empirically it kind of works, but it's a little scary. We don't even understand fully all the details here. So yeah, this is a big problem, the barrier to security.

But there's good news in that because it does not favor defenders, but I don't think it favors attackers either.

Not yet.

I think that if you— that lack of transparency kind of favors nobody.

Hold on. There's no way Anton Chuvakin is on the show right now saying a system that the defender doesn't understand how the rules work is good.

I didn't say that. I think it would be kind of not great for defenders, but the point is that it doesn't automatically give an advantage to the attackers.

Okay, I agree with that.

That hurts defenders, but it doesn't help attackers.

I agree with that.

But you can be worried that As soon as you figure out how it does work, the first person it's going to benefit is the attacker.

Yeah, okay.

Because as soon as you understand it, then the attacker starts doing something, and then you have to figure out how to defend this new thing that— oops, I realized that it understands, it works in this way, and now you have this new kind of attack.

Okay, so a mini conclusion on this one, on this part. In your opinion— and I know you think it would not be like a research-backed dataset but more of an opinion— in your opinion, this lack, relative lack of transparency, does it help Attackers more, defenders more? It helps attackers first and then they help defenders later? I don't know.

Yeah, I think it's a very hard question. I would sort of go with, if we could understand how machine learning models work, initially it might help the adversary because we could develop better attacks, but I think ultimately it will help the defenders because once we understand how a system works, then we can develop techniques to secure it much more easily. But yeah, this is mostly conjecture.

I think that's a very reasonable position to take though. It's easier to secure systems we understand.

Yes.

Yes. Okay. So I want to ask our traditional closing questions. First, do you have a weird tip to help people with their AI security? And then 2, do you have recommended reading? And 3, listeners, I will remind listeners that Nicholas is speaking at RSA and you should absolutely check out his talk. You can't mention your talk as further reading, by the way. That doesn't count.

Not going to do that. Reading? I really don't know here.

You can go with a long list of research papers. Research papers are fine.

I would cautiously recommend reading some of the most recent academic research. Some of this is inscrutable, but there are occasionally papers that are written that actually make sense. And this field is moving so fast that there are a couple of books on adversarial machine learning, but oftentimes by the time the publication even happens, the field has moved on several cycles, and so it's too late. And so if you can find papers that are interesting to read, I think I would encourage people just to do this, if only to understand at a very high level what problems they're trying to solve, because this might be— you might not understand exactly the details of how the techniques work, but understanding what problems are being considered is, I think, useful and valuable. Your second question was any weird tips? for securing systems? Here, no, I don't have anything. I guess my only tip would be don't do it. If you really need it to be secure, maybe not machine learning right now. But yeah, I don't think I have any great solutions here.

Is it a little bit in the bucket of if you want good encryption, don't write your own? Are you inclined to say if you want good AI security, don't write your own AI?

Yeah, but at least for encryption, I can turn to one of my cryptographer friends and say, what do I do here? For securing machine learning, I don't have anyone I can turn to to give me the secure machine learning thing.

Oh, that's a good point. Yeah, that's a sad but correct point.

Yeah, so maybe in 10 years we'll be in this world where there exist people who know how to train secure machine learning and will tell people— most people don't train it and go ask an expert. But even today, I don't necessarily know that that's the case.

Well, on that cheery note, Nicholas, thank you so much for joining us today. Listeners, I hope you found this as I certainly enjoyed this conversation.

Thank you for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.