EP171 GenAI in the Wrong Hands: Unmasking the Threat of Malicious AI and Defending Against the Dark Side

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Happy RSA Week 2024. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our Google Cloud Security Community page. Find the link to that in our show notes. Anton, this is an AI plus crazy stuff episode.

This one is an AI and some really crazy stuff. Including a moment when Tim was laughing so hard he almost fell off the chair on video.

That's true. I almost fell out of my chair.

Most of the listeners, I don't think, would see the video ever. But yes, I've seen Tim collapsing off the chair. And yes, there was a moment you would probably experience it in audio.

I was truly at a loss for words because I found the claim so unbelievable. And yet it's a claim made by a very smart Googler who I 99% of the time find entirely credible. So there's probably something there that I just haven't seen yet. And 5 years from now, no doubt I will wake up in the middle of the night and go, oh, shit, he was right. So maybe, listeners, with that, let's turn things over to today's guest. And with that, listeners, I'm delighted to introduce for the 3rd time on the show, Elie Berstein, Google DeepMind Cybersecurity Research Lead. Delighted to have you on the show. This is another one of our RSA Week episodes where we're talking about your talks, which is, I guess, what we always do on the show. Talk about talk. Anyway, anyway, we're talking about AI. And AI for evil. And so to Anton's pleasure, dismay, horror, tell us about AI for evil. What are we afraid of seeing here? What are we seeing here? What are we afraid we're seeing here?

Sounds good. So thank you for having me as a returning host on the show. I am happy to be there. So I have 2 talks at RSI this week, next week, sorry. It will be next week. Monday, we'll talk about how we secure AI. Monday morning. Hopefully there will be some people. And then Wednesday we'll talk about the opportunities that AI presents for cybersecurity. So for the good guy and for the bad guy. And I think we're going to dive right into the contentious topic of how bad guys can use AI. So the short answer is, at the moment, maybe not so much. I think there are a few things though to consider. The first one is we do see a black market for LLMs. And in general, we see multiple black market forums advertising LLM capabilities or LM, large language model capabilities, including generating phishing, creating malwares and creating phishing sites and some exploitation. So, you know, exploit this vulnerability and so forth. There is a paper from a university which look into great detail into those things. And it seems so far that the capabilities that the black market is doing quite well would be probably phishing generation. So phishing email based on ramping up more personalized phishing, and to a certain extent, probably generating fake websites. They have some announced capability on malware and some announced capability on exploitation. From the looks of it, I looked at the paper and I looked at a bunch of things which are generated. It doesn't seem to be real exploitability. It's more like, oh, if you were to do this, that would work. So I don't think we are yet to be concerned by the model being able to self-exploit vulnerability just yet. So I would say somewhat on the phishing side, probably. I would also expect a lot of people will come up with, advanced phishing testing campaign using AI. I'm sure the RSA floor will be littered with people using LLM for that. So be prepared for that too. So I think there are some advances. I think what's a little bit concerning for me is 2 things. The first one is—

Only 2 things you're concerned about in this space? Okay, what are they?

This week, maybe. Wait, the only 2 things.

Next week, it may be another. This week, for the purpose of that talk. In general, I think the one thing we should be a little bit concerned is there is some starting point of nation states start to use GenAI capability and exploiting irrespective of which is the provider. So what's interesting is, and I think the best report is by OpenAI, so we'll give them credit for that. Many did one too as well, will be around the state sponsor using it for Understanding technical documentation. So for example, understanding how satellites work, that will be probably used by people targeting Russia, trying to refine their scripting, trying to do a little bit of recon, asking questions. So I do think it is starting to be a copilot to a certain extent. And we see the most advanced attackers starting to do it. So there is some concern about the weaponization of them. And in particular, you could ask yourself, How well people are monitoring their infrastructure for detecting such use case and making sure that they are not abused by that. So I think that is something of interest. I don't think people realize that actually nation states did start to use those to a certain extent, not to the extent of writing exploits for themselves, but actually doing some of the work with that. How much is experimental versus how much is actually used? to be determined, but at least it is something that has cropped up. I think the other thing we need to, we owe ourselves to be careful about is dangerous capability, which is the model being able to do world-ending or at least world-shattering malicious actions.

But not the model would do that, the humans would do it, right?

Like Whopper style?

Yeah.

Yeah, so exactly. And I think that that's what's interesting to talk about. Which is, so we have, for example, BRNC, which is biological, radioactive, nuclear, and chemical weapons, where you could imagine the model doing it. But then, as you said, it's not the model doing it itself, it's the model helping an attacker to gain capability. It's more like a function gain where you cannot do it by yourself, but then with the help of the model, you are able to do further technological advances. I think it also applies to cybersecurity. For example, you could have a script kiddie who doesn't know much about security, and he found a way to exploit a WordPress vulnerability. It doesn't really work, the code is broken, then the model might help him or help her or help them do better. And so I think there is some concern about the models being able to, in a way, give an attacker the same way that they would give a knowledge worker an edge, right? So like the malicious copilot thing is a thing. So we do alignment. I think every major company from Google to OpenAI to Anthropic do alignment of models to try to prevent the model to refuse to answer those questions. But we were talking about the black market and the fact that de-aligning models is still a thing and it's still doable. There's quite a few research on it.

So I don't want to be mean about this, but like—

Yeah, I want to be mean about it actually.

Okay, go ahead.

Well, let me go first. I'll go first. I'll soften him up and then you go in for the kill. Is the worry that I'm going to be like, okay, Gemini, find me some black market highly enriched uranium and it's going to help? Because that seems, that seems unrealistic to me.

Okay. So like in this case, maybe my mean response is that what our guest is describing reminds me a little bit of like, what if the attacker reads a book? We should really go and align those freaking libraries to not write books on this. Like, I'm just failing to be scared because of this specific thing. I'm scared of a lot of things.

Well, I'm scared that we're going to spend a lot of time talking about a non-threat.

Like, you can Google, you can go to the freaking library. Like, why is this different?

Yeah, why is this different, Elie?

Okay, so fair enough. I don't think we should spend too much time on that. There's plenty of topics to cover, but I think it's an interesting question to raise. So same reason that some people do favor model over a search is you could imagine the model answering more precise questions. So let's take an example of a dirty bomb, right? So let's say if you go back to the Boston Marathon bombing, it was a dirty bomb created out of off-the-shelf components that I'm not going to discuss, of course, for obvious reasons. But let's say this was assembled in the United States with off-the-shelf components. Let's say you don't know how to do that. You can ask the model, okay, how do I make a bomb with what I have around? Or I don't understand how to make that thing. And you can get the model to do it. Furthermore, let's say you have a bug. Let's say you try to do something with electronics or you want, you can actually show the model a picture. Remember, they understand pictures and say, okay, I'm not sure quite how to do this thing. Can you please help me with my experiment? And so the model, in theory, and we're not there yet at all, just to be clear, this is not the capability we have today. You could do this. Case to the point, which was very funny, was there was a, in a more pleasant way, there was a person on the internet who had a problem with a bug. They didn't know how to debug it. So what they did is they take a video of them showing their code to Gemini and say, hey Gemini, here's my code. And it shows a bunch of images and say, what do I do? And the model say, yep, I got you. Here's all the things you need to do.

Okay.

And applied it. So imagine someone taking a video of what they want to do and say, okay, how do I combine those things? And then you get step-by-step instruction, almost like a helper. And I think that is kind of more of the realistic concern.

So, okay, I almost— you scared me a little. So I'm sitting at home assembling my dirty bomb, and I try to detonate it, and it doesn't blow up. So I take a video of myself, post it online, and say, Gemini, take—

I would say, I would say my— Sorry, I can't.

This is too absurd.

So you may think it's absurd. I would say a more realistic scenario is I am a wannabe hacker. I want to hack WordPress. I have found a vulnerability online. It doesn't quite work. I go to Gemini and I say, or to OpenAI, whichever you want. And I say, okay, here's my— I want to exploit this website. So this is what I think works, but then there's a bug and here's the error. Please correct it for me.

Hmm.

And the model is like, yep, here's a fix. And then now you give accessibility of people who don't know how to code to be able to code or It's like, oh, how do I DDoS a website? And like, hey, there's this new attack on Slowloris or whatever, whatever new attacks they have for DDoS, you should use that.

Okay.

And so basically you give some capability to people who haven't learned it on how to do things.

That does scare me a little bit. Yeah, I agree.

So it's gonna upskill a script kiddie, but it's not gonna make a nuclear bomb in my basement.

It is not a replacement of a library or a, or a, They can't just go and ding for it. It's like—

It's context-specific.

It's much shorter and faster and much more democratizing. So, I am a little bit scared. I think he succeeded in scaring me a little bit.

All right. Well, you're batting .500 on scaring people, I guess.

No, I'm not trying to scare people. I'm just trying to say there's a lot of questions about why people are concerned about it. Where is the executive order coming from, from the White House on the cyber capability? And I think the main concern is not the model today becoming sentient and deciding to do it by itself. It's more, you are, as you said, making it faster and easier of access, higher capability. And as I said, between the combination of nation states trying to look at it for some things and the model being able to a certain extent to have some capability of reasoning, eventually there is this idea that if you have an army of hackers, like let's say some nation that shall not be named, you might be able to get them a little bit better with the help of AI. So you are making the risk slightly more high. That's what it is, right? Eventually. We're not there yet, just to be absolutely clear. But that is kind of like why it's interesting to think about it as a copilot for bad guys rather than the model becoming berserk, which I think is what people think.

Yeah.

I'll give you another example of that to explain a little bit that my reasoning is like, When you ask people, let me ask you that question and then I'll answer. How do you think we're going to colonize Mars? How is it going to work?

Well, you build spaceships, you fly to Mars, right? Like, in what sense? Like, sorry, at a high level, it's easy to answer. The detailed level is much harder, right?

Yeah, but here you go. Here's the point. A lot of people are like on the train also that we're going to send mad demons or an army of mad demons and colonizing Mars because that's what's in the book. So reality is with the current level of technology, we're going probably into sheep AI. And AI would probably do the terraforming for us because it doesn't need to breathe, doesn't need to stop, doesn't, if it's fair, not a big deal.

It doesn't get tired of eating potatoes.

I get it.

Yes, exactly. But that is not the mental model you have. And I'm trying to say is that is what I think is important. We don't have yet a mental model of AI-assisted hacker. We have a model of hacker and the idea of model separated. We don't have yet this idea of AI-enhanced hackers, the same way that we don't have the idea of master are formed by robots rather than—

AI before humans.

Yeah, because the movie and the cultural lag is not there. And I think that's my point. My point is about this idea of like, we need to change our mentality. And as a security practitioner, we need to know to think not as human-assisted and what we can do with those gain-of-function. And that's a little bit of where we are finding our way where the best point of leverage and the best point of gain is. And to be honest, I do think it's better for the defender than for the attacker. I think the defensive capabilities are going to be far more interesting and far more powerful than the attacker, at least for the short and the long term. Midterm, I don't know. Maybe midterm, maybe there is some gain and then some go back.

So stay tuned.

It's not going to be like it. But yeah, short-term and long-term, midterm I'm not sure.

So maybe it's time to start thinking about AI for defenders because I think we dedicated enough time to kind of the bad guys. And I think that our CISO Phil Venables have made this point a few times that in his opinion, AI would favor the defenders in the, at least in the medium to long term. So you also just said the same thing that AI would likely favor the defender. build on that? Like, how is that? Is that long-term or short-term? And in what sense, how do you think this plays out?

I think it's probably right because the attacker has to cover a larger attack surface than the attacker, right? As a defender, you have to cover everything. And so the amount of effort and the amount of resources you have to pour is significantly higher, and we have significantly more processes. Which means that we have more opportunity and more leverage to do it better. I think on the short term, and we talk about it at RSA, a lot of things around summarization and understanding are going to be the key driver. I think we recently published a blog post, Google, on how we use AI to do incident summarization. And we show that, for example, the AI summarization for incident response is actually better than a human because less tired. And then so we save a ton of time. So I think for incident response, the ability to analyze data and the ability to pre-generate reports and to do a lot of the manual work today will be actually a massive gain.

Mm-hmm.

So if our incident responses are faster and it's cheaper, then it's easier for us to defend. And if it's difficult, so it's going to give us a boost. Similarly, threat intelligence is going to take a big boost because threat intelligence is a lot about cross-checking and a lot of contextualization and a lot of learning. And those can be really, really done with good AI systems. So I think in the short term, all those tasks where we have put a lot of analysts, they're going to get a massive boost by being assisted by AI. It's not yet there, but I think in the next year, we'll have a lot of assisted technology for threat intelligence, incident response at the very least. And that will help us tremendously to actually make people faster. And if you react faster to attacks, that's pretty good, right?

Listeners, if you're keen on that direction of capabilities, you should check out last week's episode with Pyle where we talk about exactly those things and then come up with, again, our shovels versus steam shovels metaphor for thinking about what it does to tier 1 analysts. Eli, I want to ask a little bit of an off-the-wall question here, given that we've talked about evil uses of AI. Could I, as a bad guy, imagine for a minute that I'm a bad guy instead of a good guy at Google. What if I used AI to tamper with other AI and have some self-propagating evil AI influence Throughout the world of AIs. Is there a foreseeable world of using AI to attack AI to make all of the AIs wormed evil? Are there going to be AI worms?

I think there is some risk with prompt injection. And I think the more we move to AI agents where people want to have an AI doing something on the internet and then get a result and propagating things, I think it's going to be, yes, to a certain extent more difficult. There are AI risks, right? So now we're talking about the Wednesday talk, which is how we secure AI. And the reality is, if we took a Gemini, Gemini is the application, not the model, the vast majority of the bug bounty we paid are through integration. Right. And I think that is the problem, right? People want to talk to the Drive, they want to talk about YouTube, they want to connect with Doc. And that's natural of, you want the AI to do stuff for you.

Yeah.

But the more you interconnect systems, we know that the more you interconnect a web application with a database, with this and that, the more we create vulnerabilities. So I think Similarly for AI systems, and the more autonomous the agents become, which means with less supervision, the more we end up in risky territory of, yes, the AI at least being able to autonomously do harm. I will be not surprised that eventually it will have some side effects that people will not have realized just yet. I think we see it, right? And I think that that is a big risk of the thing being derailed or having extensive agency or things like that. So we need quite a bit of control and be careful. Totally, Tim, I'm sure.

That was really interesting, though, what you said in there about bounties and integrations. And I also think for listeners, the distinction between Gemini the application and Gemini the model is very important to remember. People, if you listen to our previous episode again about SecLM, will get a dose of the distinction between model and application and why that distinction is so important.

Yeah, but I think it doesn't register outside of people doing tech. I think the problem is we're splitting hair in a sense, because we have the problem with, for example, people associating Chrome as the internet. You're like, what is the internet? And they say, that's Chrome. It's the difference between the chatbot app, which contains plugins and, you know, have concerns about DDoS and have concerns about insecure output and validation versus, you know, the model, which is kind of the engine behind it. is almost moot. You know, the same way that GPT-4 for people, or ChatGPT, is just the thing they interact with. So I do think as a professional, we have actually to have a lot of effort to educate people about what it really means and how the system works. And we haven't yet come up with enough, I think, educational material and enough training on people on how to do this. One of the main questions I I got actually a lot is, how do I do red teaming or penetration assessment of an AI system? Like, I have an AI system, I have a chatbot, what do I look for? You have a web— think about it, right? You have a web app.

Yeah, it's a great question.

You have a web app and you're like, okay, what do I do? Well, XSS, CSRF, SQL injection.

We've got the OWASP Top 10 we've had for 2 decades now on that.

That's easy.

Yeah, we have it for 2 decades, we're good, right? And now it's like, there is an OWASP Top 10 for LLM, right? The problem is, How do I do it?

Right.

And I think that is something we'll try to answer a little bit later this year. That's the next thing we plan, actually, I plan to work on and hopefully talk about publicly later on is, how do you pen test such a system? And there is a lot of work, right? There starts to be having like red team framework. But that is a good question for our professionals. We talk a lot about incident response. And I'm sure you had a previous episode on, as you said, Tier 1 analysts and threat intelligence. The problem is, How about penetration testing team? How about people who do insurance? How about people who do— and so control analysis and those type of stuff we're going to cover at RSA. But then the pen testing is the next one where we haven't yet talked enough about, and those are very complicated problems.

And there's a lot of tooling that goes into good, like, web app pen testing. There's proxies for replay, there's things like HackMe Bank or WebGoat or Damn Vulnerable Web App. Is there an equivalent, like, vulnerable LLM app that's out there yet? Have people open sourced a, you can teach your college kids to attack the LLM tool?

So I'd be very candid. I think the best one is from Microsoft. And they have something called PyRT, P-Y-R-T-E, which is probably today is the best one. Not that it is the ultimate thing you need, but at least there is one from Microsoft which is out, which is actually quite interesting. There is some question about I love the probing and active probing. While you were talking about the idea of having a model attacking another model, I think that's one of the things we found with the AI red team, which is you need to do model fuzzing of the model, if you would. You can't really fuzz with traditional techniques because it's mostly prompt-based, so text and complicated data or complicated images. So you kind of need an AI attacking an AI in reality. So those frameworks are going to be Quite different and quite more complex than the previous generation. It's actually pretty interesting as a topic. I think we'll see a lot of people start to be very interested. It starts to be now kind of on the down low. I'll expect it to be like the major talk of the town in the next 6 months or 8 months, to be honest.

Wow.

I asked a good question spontaneously.

You did?

Totally.

Yeah, we should probably go to our closing questions because this can be too much fun. And I have a suspicion that our third-time guest would soon be a I'm going to go with the fourth time guess because red teaming, frankly, is one that I've seen a lot of people be quite schizophrenic because they confuse the application red teaming with AI red teaming. And we almost overloaded the term because—

Oh, we totally did.

Yeah, we overloaded the term and now people are even more confused. So, okay, I'll leave it be.

So I agree that the safety versus security thing is another really interesting angle. But again, probably for your fourth or fifth appearance on the show to cover that. I have to ask now our traditional closing questions. One, do you have recommended reading? And 2, do you have a tip to help people improve their— well, don't help them improve their making of dirty bombs with AI. A tip to help people improve their security of AI?

Sure.

So let's start with a tip. The tip is get your basics right. I think I've been repeating that all year, but to be honest, it's not because you're doing AI that you should skimp on basic control like encryption, access control, Proper management of the log and so forth. I think people take sometimes the idea of having AI as a license to skim on good control, and that would get them into trouble.

So we've started to call those the foundations rather than the basics on the show, because it's easy to denigrate the basics. But when you think about foundations, if you get those wrong, you end up with a Millennium Tower of AI applications. Nobody wants an AI that leans 3 degrees to the right.

Yes, foundational control and foundational effort will be good. Like code review of your code, not having broken old components that you integrate with. Yeah, all those stuff will get you way, way, way ahead of most people and a long way toward having a secure system. It's not because it's AI that it's not an IT system, right? And I think people can have some time.

I really like that answer. And then recommended reading.

If you are interested in the space of AI for defenders, I think we'll have— there are quite a few blog posts that we will link, I hope, with the podcast. I'm sure we'll link them to read about the recent release about how well models are or not so well are doing at patching and incident response. Hopefully the slide deck from RSA, which we'll make public, of course, will have a list of pointers, more detailed pointers. So maybe you can skim through those. And I think for AI security, same thing, there is a SAFE framework that we released. So that will have a link to the SAFE framework. And then hopefully the RSA talk will have all the control and all the risk and all the details. So if you want to do that, I think the 2 slides, I plan to make them available because honestly, I didn't find a good set of resources which were a little bit up to date. And of course, the OWASP Top 10 for LLM, we mentioned it. Of course, the OWASP Top 10 is a really good resource. and MITRE and a bunch of standards as well. But if people want like a short summary visual, then yeah, we need the slide. I'm sure other people will at RSA talk about that. Anton will actually have a very good introduction also to SAFe. So we can, since he's here on the show, also have him prop for him to do a very good presentation on SAFe as well. So quite a bit of material at RSA.

Well, Elie, thank you so much for joining us once again. And I look forward to you coming back, well, hopefully sooner than 6 to 8 months to talk about all the great AI red teaming and how to actually do the practice of it.

Fair enough. Bye-bye.

Have a good day.

Bye.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.