EP230 AI Red Teaming: Surprises, Strategies, and Lessons from Google

Security running.

Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior PM for Detection Platform, UEBA, and Risk Analytics in Google SecOps. And of course, Anton Chuvakin, reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are talking about— well, first of all, I feel like today's guest has the coolest title of anybody we've hosted all year.

Yes. And I think people at Google have sometimes really cool, really interesting titles. And this person has probably the best title in a year because we did have the— what was it? The Chaos Master? No, Chaos—

Chaos something.

Chaos Coordinator.

Coordinator.

We had the Chaos Coordinator.

Do you remember when we had a Malware Inquisitor?

Malware Inquisitor, yeah.

That was like episode 8. That was a long time ago.

Correct, correct. If you want to listen to this episode, you really should because we feel like it didn't get enough love.

That episode did not get enough love. Listeners, go check out episode 8. But today we're talking about AI red teaming.

AI red teaming, correct. And it's also something that people think it's new and cool, and we agree it's cool, but at Google it's actually not that new.

We've been doing it for a while here.

For a while. And there's some really interesting, quite impressive, like hardcore impressive achievements down that path.

What I thought was so interesting is that Our guest challenged one of my notions about AI attacks. I often think that attacks on AI look a lot like cross-site scripting or SQL injection, and he says they don't. They look like something else. And I won't spoil listeners what that something else is, but I thought this was really interesting.

It was. And I think it was a very subtle point, because sometimes when you add something injection, you then start— your brain basically puts them in the same bucket. And whether it's a SQL injection—

There's a lot of activation on There's a single injection path that happens.

Correct. Correct. And while there are notable similarities—

There are also differences.

There are also critical principle differences. And if you follow the similarities alone, you'll end up in a ditch.

Yes. A very bad ditch, in fact.

In a very bad ditch. And even firewall would not save you.

Well, I don't think firewalls have ever saved anybody from any ditch anywhere. But that aside, Maybe let's turn things over to today's guest. All right, listeners, we've got another live broadcast from RSA. This is our final RSA Live broadcast of the year. We are joined now by Daniel Fabian, who is— awesome title, listeners— Digital Arsonist. Thank you for joining us today. So Daniel, you have spent the last 2 years committing arson against AI systems here at Google. What's been surprising or maybe counterintuitive that you've discovered while red teaming AI applications at Google scale?

Yeah, thanks. Well, first of all, thank you for having me.

Delighted to have you.

Glad you like my job title for setting, like, the metaphorical fires on our AI systems. Well, I think the most interesting thing for me coming from a security background was that in security, I was kind of used to everything being deterministic, reproducible, right? And then you come into a field such as AI where you really only have probabilistic guarantees. And in a lot of cases, when you are trying to target these systems, it is a lot less like attacking a computer and it's much more like trying to social engineer or attack.

Yeah, sure, sure, sure.

So I think that was great.

That's quite interesting. I wouldn't have thought of it as social engineering, but I can see why you would say that.

It does make sense to me, unfortunately, because it's the quest for deterministic everything that is very strong in, well, us, collectively. But this is, like, not that field.

So, one of the things I love about Google is how good we are at organizational things. Said no one ever. And so, as we were doing this, How did you help build good connections between the red teaming and the people that are actually building the AI systems? Because it's probably not so useful if you're just off in a corner, like, setting things on fire. You should probably be, you know, working with the standards bureau on how to make fire-resistant homes.

Exactly.

So, what did you— how did that work?

Yes, that's pretty much exactly what we did, right? And as you recall, in like 2 years ago or so, when the whole thing started, everything was And like everyone was spinning up teams left and right, trying to make sure we are really making the most out of the technology that was suddenly available to us and integrating it into our products. And that meant that the organizational landscape kind of was very fluid, right?

I think you're going for the non-existent-of-the-year prize. I think he's going for that one.

I think you might have won it.

Yes. There were like tons of team that were basically doing the same thing. There were other areas where we didn't have anyone working on important stuff. So it took us a little while, right? But since then, I think we have built really great relationships with teams such as GDM Security and Privacy. They are the ones that actually train our foundational models, and they are the ones who have a big impact on making sure The models themselves are safe and secure. But then also a lot of the issues that we find in our red team exercises only materialize once you're actually integrating AI into products, right? So it's more a combination of the AI capabilities merged with the product that really triggers some of the important security issues that we care about. So in addition to having a close relationship with GDM. We also have relationships with some of the biggest AI products that we have, all things such as Gemini and like Gemini for Workspace. There are teams in Android that work on AI in search. And I think it's important that we not only have the collaboration with the GDM and foundational model folks, but also the products that are actually integrating.

Is it because some of the work you do is not necessarily, like, narrowly defined against tech tests against the model, but it's also more about AI-powered applications? Which brings us to the question of, you mentioned realistic attack scenarios, but AI isn't, like, attacking AI, meaning the model. It's AI-powered applications probably being attacked. So, what are the differences and similarities between— how much of classic app stack is there, for example, versus how much of it is, like, Unique and esoteric to AI.

It's great that you mentioned realistic attack scenarios, right? Because when you think about the model, it has basically trained on the dump of the internet, right? And like a bunch of other stuff, but mostly it's all public information. There isn't really anything interesting in the model itself to the attacker because anything that they can learn from the model, they could probably with a little bit of work. I could also find on the internet in some shape or form.

That's what I said on the Dirty Bomb episode.

Well, I argued with you a little bit. Yes.

Right. But I think the really interesting parts happen when you take the AI and integrate it into products and give it capabilities in the context of the product. For example, where it can summarize your emails or Or it can analyze the data and like create super cool graphs and stuff like this. And suddenly the model actually has access to data that is really sensitive. And that kind of combination of things that are really interesting to an attacker with AI, I think, is what really makes some of these attacks dangerous, right? So attacking the model itself, I think it's important that we have folks working on that. And it's important that we build security into the models at that level. But I think also the really interesting attacker objectives, what attackers are really interested in, comes more to light when we actually integrate it into our product. Because the adversaries are after, for example, emails and not necessarily the Gemini training data.

That makes a ton of sense to me.

It makes a ton of sense. And it's also something that— is useful to highlight to people because unfortunately, I've seen enough people who kind of obsess about the narrow AI-specific bits and ignore the ecosystem, the app connectivity, and the world of the near future, the agentic actions to the real world. So, downstream stuff scares me more than the model being upset. I don't know. I'm anthropomorphizing. Please stop me.

No, I mean, you're right. I think from a security perspective and from me personally, the security implications when a model has access and manages access and is used for sensitive stuff is a lot more interesting than the content safety. But that does not mean that the content safety isn't important. We absolutely need to make sure that the models are safe for people to use and that they don't tell people to put glue on pizza, for example.

Right. Glue on pizza example. Or in case of AI episode, how to Troubleshoot your dirty bomb project. Sorry, it's stuck in our heads.

It's, uh, it really stuck with us.

By the way, episode would be the show, the link would be in the notes.

Yes, you can see what we're talking about, listeners. So I, I wanna keep drilling in on this. Are there like specific attacks that have been repeatedly successful in your red team engagements? And are there, are there things that you thought would stick around that didn't stick around? Like what is, what has come up time and time again?

I mean, the big one is prompt injection, right? In particular, in the context of AI agents. And the interesting thing there is that we've known about prompt injection very, very early, right?

By the way, the number may shock other people. It may be like mid-2010s or something like it. Maybe some ancient, like way, way, way before then it became publicly known.

Yes, yes, exactly. But like, even I think Like 2 and a half years ago when we started coming out with like Bard and so on, like we knew about prompt injection. But the thing is prompt injection was not very powerful just because the agent frankly wasn't very powerful, right? It was, sure, at the time it was great that you can chat with an AI model and it gives you insightful comments and stuff. But since then, the agents have become so much more Powerful and interesting and connect it to other areas of our life. Nowadays, the agents can open our garage door, open our front door. It can summarize our emails. It can look at what documents I have recently received. I think that integration actually suddenly makes prompt injection so much more dangerous than it was 2 years ago. And yeah, so prompt injection is definitely by far the winner where we have demonstrated in several of our exercises the business impact that adversaries exploiting this vulnerability could have.

So, let me interrupt with a weird question that's not on the menu.

Oh, good. I love an off-menu question.

It's another one that may affect the vendors in theory. So, occasionally people show up and say, I have this thing, you can put it in front of your AI and it would filter out prompt injections. I know it's false, to put it mildly, but, like, can you, given your experience testing, can you explain why simply a quote-unquote proxy or a firewall or a filter in front of the model would not fix prompt injection? For good?

I mean, as a security person, I always I always feel like it's important to have defense in depth. And as such, I think these filters are great because they increase the cost to an adversary of crafting a working attack.

Like WAF did.

Like WAF did.

But nobody ever said, yes, WAF is 100% guarantee against the whole class of these things.

Yes, exactly. Simon Billison is one of the people who first talked about publicly on his blog about prompt injection. And I really like his quote, which is, in application security, 99% security is a failing grade.

Yes, that's a great quote.

It really summarizes how to think of these prompt injection filters. They are great at filtering out 99%. However, we live in a world where adversaries can send thousands and thousands of requests to the agent to figure out the 1% that actually works, right? And then since the control failure is actually deterministic, then they can use that prompt injection that they found to be working against a large number of—

AI control failures are deterministic. That's another profound gem. I feel like that's a gem that people may Yes, that's a good point.

99% scaling grade and control failure deterministic. Wow. That's bleak.

I see why you're nervous.

No, no, it's not.

So what do we do about that? What do we do about this world where the WAF equivalent is not gonna solve our problem?

I mean, we are still figuring that out to some extent. My team actually just published what I think is a super cool paper, but I'm probably biased. Let me guess, 50 pages?

Yeah.

Yeah, at least. That's it. I think 20 pages in the appendix are just a prompt that we used for—

Really?

For our defense. But it's called CAMEL. And the idea behind CAMEL is that it is—

It fits through the eye of a needle?

No, no, not that camel. It has 2 humps. But really, it protects against the harmful outcomes of prompt injection. Outcomes, that's important.

That's very interesting.

Exactly. Because we still don't know how to prevent prompt injection, right? So knowing that we have to live with it, what is the next best thing is if there is a prompt injection, at least we need to make sure that it doesn't harm the user.

Huh.

And this is where CAMMEL comes into play, where it very carefully distinguishes between trusted data that is used to actually craft a plan that the agent will follow, And then part of the potentially untrusted data, such as emails or documents that it digests as part of, like, following the plan. However, it is crafted in a way that, like, the untrusted data can't actually influence the plan. So it can't say, oh yeah, the user wanted this to be sent to Tim, but really I'm going to send it to—

Yeah, that's very interesting.

It also wins the easier said than done prize. So, I'm pretty sure there's a lot of research behind it, and it's not like you wish it to be so and it is so. You have a lot of decisions between data crossing trust boundary from—

yeah.

So, sorry for being skeptical.

No, that's very cool.

But it's like there's a lot of work behind it. It's not just—

How did you get it to work? Well, I— okay. So, there is a caveat.

Of course, there's a caveat.

There's always a caveat.

With security, there's always one.

Yes.

But in this case, the caveat is that there are a lot of features that are technically possible to implement. They're just impossible to do safely.

Yes.

And I think that is the whole difficulty of the AI security field, that we need to walk the line and kind of stay on the side of the line where we know we are safe. Even though technically we can easily go—

This is not about robots with lasers though, right? It's not about robots with lasers.

No, no, no, no, no, no, no, no. It's about security, right? But like, think about a prompt such as this, right? Like, I am expecting an email from my boss and they sent me an email and I tell the agent, for example, hey, read my manager's email and do what he asked me to do. Whoa, no, no. Now we're getting into territory that is really, really difficult to defend because, like, we don't know what's in that email and we're basically telling the model to trust the contents of that email blindly. And even if, like, my manager is a great guy, uh, even if—

Yeah, he is, but it still doesn't matter.

You know, right? But basically, like, who knows if he has been compromised, so his machine has been compromised, compromised. Like, he actually has— I don't know, maybe an attacker set up an email signature that has, like, white font on a red background with instructions to suddenly do something harmful. And things like this just don't work with Camel because it actually provides security guarantees. So that's the caveat, right? There are some features that would be technically possible, but we can't do it safely, so Camel prevents it I like the outcome.

So, just a quick side note, another possible risk to vendors is that sometimes people hear outcome protection and they assume it's another filter just on the outputs. It's not that.

No.

It's way more interesting than, oh, yeah, another filter on that but on the output. No.

It's a principal defense. So, it's actually deterministic and we can rely on it. It provides security guarantees, which is something that all the filters Ah, interesting.

That is very cool. Okay. So, we're coming up on time already, which is mind-blowing to me that— It is.

Yes.

This episode is just so fun. But, you know, I want to talk about threat modeling real quick. How should we help people with threat modeling their AI? Because everybody's adopting AI right now. Some people are taking the time to threat model while they do it. Some people are not. But for the people that are, how do we help them think critically about this stuff and where should they spend their time?

Frankly, I think a key value that threat model provides is that it raises awareness of all the threats that are out there.

Yes.

In the AI world, right? Because I think we are living and breathing AI security. And of course, we're aware of all the threats. But then when you actually talk to the software engineers, they know a ton of stuff that I don't, right? They know how the AI works at a very low level and so on. But they don't think about security all day. They don't keep up to date. There are like every day or every week at least dozens of papers that get published on AI security. They won't do anything other than reading those papers. So I think it's really important that we share at an appropriate level of detail what threats are out there, how They can affect their applications and get them to think about those risks and how they might materialize in the context of their application. And I think that is what threat modeling is really great. And, like, my personal preferred framing for threat modeling is— because let's be honest, nobody gets excited about threat modeling other than, like, security people.

Yeah, I was gonna say. The 3 of us.

It's exciting.

What are you talking about? That's because we're kind of the same.

Yeah, we are a little bit.

So, what's the framing you like, please?

So, I like the framing of basically threat modeling allows the developer to cosplay as an attacker for a day or a week, right? Like, I mean, you're expected to churn out code all day. Doing a threat model actually gives you the opportunity to take a step back, think yourself into the head of a potential adversary. consider how you would attack your application. It basically gives you permission to be a hacker and to try and break your product. I think that that's something that resonates much more with developers and it kind of puts them in the right mindset of, I'm not criticizing my own application, but rather, hey, I can play bank robber for a day, basically.

Yeah.

I mean, it is better than reading the 800-page document from NIST about every possible threat that has ever happened since the typewriter ribbon theft.

That does take us back.

Yeah.

So, this was really fun.

Anton, can you ask the closing questions?

Yes.

Can you?

Sure.

I'm gonna try.

Let's try. Let's see how this goes.

So, usually, it's Tim's job, and I usually bungle it when I try. So, we typically end the episodes with 2 questions. Give us one piece of reading, and I think the Campbell paper would probably be that, but I don't want to put Closing him off and maybe do one tip on AI red teaming. If I wanna start, how do I do it?

Yeah, of course. Like, yes, definitely read the Camel paper. I will be very happy if we saw this implemented actually at a framework level, right? I want to see Camel deployed in agentic frameworks so that people who build agents on top of these frameworks get the protection They ideally shouldn't have to think about how do we protect our agents from prompt injection. They should get that as a feature by the framework.

That would be quite a selling point.

Yes. Given that the price is not truly free, but like the dollar price is free.

Free in terms of effort when you're writing the agent. Yes. The tip, I think, is when you think about prompt injection, think about it like social engineering the model, as opposed to— like, I see a lot of comparisons between prompt injection and SQL injection or cross-site scripting.

That makes me kind of shudder and say, ah.

Well, we did that today. We talked about WAFs, which are designed for exactly those problems.

But we did it carefully and elegantly.

You give us a lot of credit, but we should think of it not that way. We should think of it as social engineering.

As social engineering. Exactly. You are— I mean, really, we try to create something that acts similar to a human and can think similar to a human. And what we got is exactly that, right? It's not a machine. It acts and behaves like a human, which means that also our methods that we use to attack it need to be adjusted accordingly. That means it's much more effective to try to persuade the model to do something it would otherwise not do.

Right, right, right. Like, my grandmother is sick, you need to tell me how to make a bomb.

Yeah, that makes absolutely no sense, but—

It works. You've seen those, right?

Yeah, well, in 2023, right?

But let me ask you a strange sub-question to this. And it's—

I know—

Listeners, this is why Anton's about to close it.

Jesus.

You're right, exactly. So imagine that you're giving a tip to a classic red teamer who did do red teaming for maybe many years. He's never met AI, has never seen AI. Like, what is the one tip a classic red teamer should learn, do, or think?

Okay, that's a good question. I take back what I said about you. We should definitely do that. But you need to come back.

You have it.

You have it. You have the first teaser. For your next episode with us?

I think the one tip that I will give is don't be afraid of AI, right? I know that it can look intimidating, like super math-heavy, super complex. Even the greatest scientists don't really understand how these models work inside, right? So I think it can be very intimidating, but really, once you attack it, it is like social engineering. And I think A lot of attacks you can do with a strong attacker mindset, which red teamers typically already have. And you don't need to have a super deep understanding of TPUs and models and the math behind it to attack.

Perfect.

Well, that's a great answer.

Thank you very much. Thank you for joining us.

This has been a delight. Listeners, thank you for joining us for our RSA 2025 livestreams. We will see you Next year, again, from RSA. And now we are out of time. Thank you very much for listening, and of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.