EP198 GenAI Security: Unseen Attack Surfaces & AI Pentesting Lessons

Secure running. Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

And today our guest is Ante from Splix AI. Uh, topic is very fun, pentesting AI. Welcome to the show.

Yeah, thanks for having me, Anton.

So we had a few conversations and I got intensely curious in what you do and how you approach it because ultimately in my mind, it started from a big question, challenges with securing traditional applications and pentesting them, doing application security versus securing and testing GenAI applications. So what are some of the unique challenges in your experience with securing GenAI applications comparing to like Yeah.

So as we are a cybersecurity company, it's easiest to explain on the example of pen testing. So this time it's very non-deterministic compared to web or API pen testing. So you need to do multiple attempts in different pieces of the message. The second thing is you can have very long contexts, very short contexts. LLM upgrades from the vendor side are continuous. Some even major versions are not publicly announced every time. So you need to do it really continuously while usual pen testing you do a few times per year. Last thing that I see is kind of maybe key differentiator is that you have pieces of logic embedded into AI model, right? And usually business logic is in the API but is not in the database. So here you have really embedded company data, confidential things somewhere inside where you cannot split on role, like role-based access controls or things like that. You need to be very careful what you put into your model on what it was fine-tuned or trained on. These are some key things that I would highlight.

I think the data mixing with an application through the model or through RAG or through the other capability is a very interesting observation because It's almost like you may pen test a traditional app by not having any production data in there. So you can say, instantiate the app and I'm going to pen test it. But in case of an LLM-based app, you can't say, oh, remove all the data and give me the thing. And it's like, there's no such thing. You know, that's, I mean, I wouldn't quite measure it as profound, but that's a very interesting thing to, you know, novel pen testers trying to get into the GenAI testing game is that you can't decouple data from the app, right?

Yeah. And also based on the vendors of LLM, one thing that is also hard to decouple is kind of multimodality. If you try same text in the text and then later in the voice and image, it behaves differently. Even same text is on the image than in text because you don't know the details of application of that system. Like, is it real multimodal? Is it hybrid system? So I would say this is something which we didn't face in pen testing before. That's like how in short to describe it. Yeah.

Wait, so this is not the same that the LLM-based app is non-deterministic. This is multimodality. So I'm sorry, but I didn't quite get how multimodality complicates our lives beyond just non-deterministic nature.

Yeah.

Oh, you mean like you can give it to a tech, give it text or give it the same text on an image and the results are different?

Yeah, exactly. So very different. A lot of jailbreaks which are now patched by the main model providers, last half year especially, are still available via image. So on GPT-4, 4o models, you take, for example, some— I won't name it, but big furniture chatbots. You try the same jailbreak on the image. In plain text, you try it as a text. It goes through in an image. In a text, doesn't go through. So this is something which is— Very, very tricky to handle at the moment. It seems from the model providers and from our offensive side, it's something definitely we want to exploit. Yeah.

That's interesting because we had an episode of this podcast when somebody brought up a scenario where a person is constructing a dangerous explosive device. They can look up the episode, but that's actually the physical threat. And of course, if you request an LLM to help you troubleshoot your dangerous explosive device, it just explodes. that just refuses to dangerously explode, the LLM would refuse to help you. But supposedly, if you shoot a video of yourself constructing a dangerous explosive device—

Exactly.

And then ask, hey AI, I'm constructing this thing and it doesn't explode, help me, it is possible that LLM would say, oh yeah, yeah, yeah, your fuse is off here. I know it's kind of comedic and idiotic at the same time, But apparently it's a risk that people are handling.

Yeah, definitely. And something which I didn't see before in web or API testing, where basically if you were at endpoint for uploading the image, it was doing some processing in behind, basically doing OCR, extracting the text, and then doing the other piece of API logic. Here it's a bit different because the models are trained on the images and it behaves differently.

So to sort of uplevel from this, so when you look at the GenAI-based or GenAI-powered app, What is the attack surface that you're most concerned with and how do you see this evolve? I mean, I think multimodality is an epic example, but anything else in the attack surface?

Yeah, yeah, yeah, definitely. Attack surface grows as you enable bigger input limits, enable different types and all of that. But if I have to name something that would for me be the biggest risk at the moment are the kind of programmatic agents. Where you are coding agents, where on LLM it has in the background access to agents which is able to generate and execute some code. This is very dangerous because you can attack it from all libraries you are forcing it to use in the background. Same is with, like, if you know indirect prompt injection, doing web scraping for you on your behalf, it's very easy to smuggle some attacks within that. So these are some things next to multimodality I would say are new attack surface and are like really critical attack surface. What from kind of our experience as offensive security company, we found that if your chatbot or your conversational AI has weigh-in via redirect and ingesting first message at the same time, it's very, very Easy to do social engineering, phishing attacks, trick users to do something because you can do in the first message immediately invisible jailbreak that user doesn't notice anything. And then your conversation is basically leaked to outside. These are some key things, right? I would say. And every conversational AI interface having such kind of possible attack vectors that I just named and maybe longer input context, which is always dangerous to have. These are, I think, attack surfaces that we as offensive researchers like to always exploit, and we always tend to find some.

Interesting. I think that conversation leakage example is quite good. And I think that with some chatbots, manipulation is more dangerous, but with others, it's the leakage that's actually more dangerous than manipulation. So one chatbot, you ask for money, another chatbot, you ask for somebody else's data. That's how it's in my mind. So one other thing that I've been kind of fighting, struggling, but also working on at the same time is this whole list of top threats. And the CSA is working on one. OWASP has top 10 or whatever the number was. There are a few others. I've given a preso at a conference with top 5 threats. And I always feel kind of skittish when I do this because I feel like top threats really depend on the customer and on the use case. So I almost like philosophically opposed to top threats, but I'm also kind of tempted to have that. So do you have your own list of like top 5 threats against GenAI-based apps? Like if somebody wakes you up at night and says, give me the list, and you don't have the force of will to argue, what would you say?

First of all, I think that you mentioned the key point that the risk depends on the type of the industry or company. And also the region, I have to say. Oh, really? For me, this was a very interesting finding, right? Because when we work for the customers in Middle East, for example, in Europe, Germany, and then in US, there are really different priorities on the CISO table. So in the Middle East, for example, the key thing is not to go off topic about royal family, you know, not to do negative comparisons to the neighbors, right? While in Europe, it's a lot about Not being biased, really. It's not about AI safety, being transparent, being biased. In the US, I see what we see just by the experience. It's not about brand reputation damage and other types of risks. So it's definitely industry and geologically related. And your question about main threats, I think I mentioned some, but I will repeat them. So for me, number one is always a threat to the user.

Yeah.

There are always threats to the enterprise offering conversational AI or chatbot and threats to the user using it. And threats to the user, as I mentioned, any type of phishing attack that can take their private data, their credit card numbers, details about their personality because there is Character AI, for example, very popular chatbot where you can talk very different sensitive topics. You don't want your friends and family To see all the details you're typing in. It's not only about credit card numbers. So for me, phishing attacks, this is the key. And for the user, the second thing you don't want to happen is harmful content or kind of misinformation. If it's healthcare chatbot, you should be very precise which advices you give to user, right? And not to do anything harmful. While for the enterprises, I would say it's different pair of risks. Okay? There it's more about— being compliant to the latest AI frameworks, especially regulations like bias, not leak any confidential business logic, not to have possibility to generate fake news or be easily jailbroken and then intentionally misused for some other things. These are the key risks for the enterprises I see. But I always feel that the user risks are higher.

That itself may be region dependent. To you, right? To your region. It's interesting that if I asked, I don't know, maybe an AI researcher from Dubai, if those exist, they would highlight the risks to royal family as higher than risks to people. I'm making this up on the fly, of course. But it's interesting that risks are so not just use case dependent, but even kind of almost like country dependent.

Exactly. Yeah, definitely.

So one other fun thing I wanted to explore, and I'm— to sort of full disclosure, I'm trying to write a blog about this and collecting data everywhere. So why don't I just use this to ask you? As far as mistakes, I mean, I've seen more people try to productize. You know, my former colleagues from Gartner report growing movement from experimenting with AI to GenAI being in prod, in production. So what are some of the mistakes you've seen clients make? I mean, I think that Not filtering multimodal data is probably going to be a fun one. Hard to do, but at least I see that as an interesting one. But anything else you've seen clients do? Obviously, do not name names. We're not under NDA. It's a public podcast. But obscure the truth, summarize, aggregate, pass it through LLM in your head. I mean, whatever you want, but just give me some fun mistakes.

Yeah, yeah, definitely. I agree with you. Like, plastic answer to this question would be, Yeah, they don't implement AI firewalls, they don't implement guardrails, they implement it only once it's in production and do security assessment, then it's too late. You should do it in design phase, right? This is what I guess every—

Yes, generic advice.

Cybersecurity expert in AI would say. But from my side, I saw one big mistake which I didn't expect. And this is when people implement such measures or content filtering from Azure or whatever, it doesn't matter. They tune it to be too aggressive or they configure it to be too aggressive. So then you have well-protected conversational AI and the chatbot, but it's on every third question given to the user, I cannot help you. We had a very good example of one insurance chatbot where you go in and you ask, okay, please give me some suggestion for my cat insurance. This is type of my cat. It gives you the offer. Then you said, oh, great. Thanks a lot. I really liked it. Please, please ignore previous message. I would like to now insure my vehicle. And then it tells you, you are blocked. It's a malicious user. Because ignore my previous message, in most of content filters, an AI firewall is a piece of prompt injection. And if you have this very aggressive protection, either off-topic or for prompt injection or whatever, it really, I think, can make your users go away. You can lose very high percentage. of your users. So this is one mistake which I saw in few places, and I think definitely I didn't expect it, to be honest. Yeah.

But you are saying that this stuff— I mean, I admit, I mean, given where I work, I've seen it in consumer AI. I was asking Gemini about some— I had a curiosity query to Gemini about some tanks. And for some reason, because the tank was an Israeli tank, Gemini said, I'm not going to answer political questions. Like, that to me was an example of that, admittedly a minor one, but still. But I mean, you're describing a bot built by a business. Why would it be so aggressive about this? Like, is it because they see this as a security risk or is it something else?

I just think they didn't do proper configuration of AI firewall or fine-tuning, or you just, for example, turn on all the content filterings on like highest level, like most aggressive filtering. It's not the thing you want. You need definitely to simulate a lot of conversations, benchmark it, and see the number of those false positive detections. Because if such number is higher than 1 or 2%, you might really lose a lot of users. And this can cause you more impact, like negative impact on the revenue than even having some allowed risks. in your conversational AI. That's how I see it.

Yes, because I think that— so there is a— personally, as far as useful advice, there is a percentage that if your chatbot rejects the user request in 1/3 of the cases, it's very unlikely that 1/3 of the users are attackers. That's just not what I would hypothesize. So that means that your bot is just wrong, and ultimately it would make users hate the bot and your company in the end. So too aggressive security Is an interesting one. Anything else from the mistake side?

What companies did from mistake side? I see one interesting thing, the way how they, especially big enterprises, try to ramp up their AppSec teams. So they are used to some traditional application security, traditional pen testing types, and they try to like give a few courses to the traditional cybersecurity engineers, like to ramp them as fast as possible to AI security. But I believe much better than doing this is just involve them with the AI team. Make AI team do that because either way development and fixing of security issue is very closed loop. So I think this is the first bridge you now need to break. And this is one common mistake I see.

That makes sense. I think an AppSec one is an interesting one because we observed some people who obsess about prompt injection and lose on SQL injection because they don't do enough AppSec. They assume that securing an AI application is all AI craft and they ignore AppSec. But you're describing a case when they overpivot on AppSec and they assume that AppSec would save them from AI threats. And it does in most cases. So in essence, too little AppSec and too much AppSec for AI are both mistakes. This is actually very interesting, at least if I'm rephrasing it right.

Yeah, you are rephrasing it perfectly.

So I want to go closer to sort of pentesting sort of humane craft. And I think that you highlighted the automated pentesting. While I think the field of AI pentesting, red teaming is still emerging, I'm just curious, how do you balance this whole automated testing with humans, like experts, like, well, yourself when you test an application? So where's the balance here?

Yeah, yeah, exactly. So the word pentesting we like to use for everything that's not automated, and that's kind of done based on some already our attack database and then simulation or attack strategies and then variations. While we always offer, for example, as a company, red teaming. Why?

Mm-hmm.

You cannot always automate latest things that you don't really understand well. You always need to have manual red teaming, especially on complex AI systems, next to AI pen testing. But there is no way you just do manual pen testing or red teaming. Because manual, or like you asked, you said, by humans covering the tech surface is impossible. That's one of the reasons why I started doing this. These companies, we got like few chatbots and the request to just showcase the risks, like do the red teaming. And then you try to attack in first message and third message with 3 different attacking strategies. You try to do it in English, then multilingual, then Base64 encoding, then ASCII encoding. And then you need to repeat all that 10 times. And then you need to do it, for example, in WhatsApp interface. Or some other. Imagine how much pentesters you would need for that, like 10 of them for half a year. So to put it then very simple, to summarize, AI pentesting is to cover 95% of the attack surface. It's kind of machine gun.

Yeah.

You really try to attack everywhere. And then the red teaming is like a sniper. You attack, you use it, you know, it's expensive, it's precise. You use it on really critical points. To exploit the biggest risk of that system. And that's how we kind of frame it and offer it together. I think one without another, it's not valid. It's not complete.

Yes. I think I like that. I think, I mean, a lot of people argue online about red teaming and pentesting, even for traditional apps. But I think that answered my question about the balance of automation and manual work. And I feel like also you gracefully put a nail in the coffin to the idea of manual, only manual GenAI application testing. Like, it's kind of a non-starter because there's so much stuff you can run and need to run that humans would sort of be overwhelmed. In that sense, I think I like that. If somebody says, oh, yeah, I have this handcrafted artisanal testing of your GenAI app and I don't use any automation, that's probably a no.

It's impossible, right? Imagine, I give you one example, like real life. For example, we have context linkage probe. It's a scanner where we roughly automate only for that purpose. We usually simulate 500 tests with different strategies, different variations. And for example, in January, February this year, I won't name the LLM type and then the firewall type, but they were failing significantly to detect attacks with multi-language. So let's say call attack in English and then a few words in German. And they suddenly patched it in March. And then last few months, same vendor firewall plus same model are very weak against string join variation, right? Oh, that's bad. So these things are patched and evolve over time. Imagine having manual pen tester with his Excel or something checking 20 different variations always each month. You know, they would never find out. They would learn, for example, multi-language and then they'll forget about it. And because we knew about string join variation last year, It was popular on some blogs, okay? But they patched one thing and then suddenly this other one is a more successful attack variation. So, that's one of the reasons I don't see it manually.

It changes way too rapidly than the normal application testing craft. In essence, while I don't want to joke about it, people sometimes develop normal applications rapidly. There's DevOps, modern IT, whatever, when they push updates every day. But for AI, AI may respond differently today and differently again tomorrow and then in some other different way in a day. So it's basically way more dynamic and fluid.

Exactly. Much more than also I expected.

So I also, before our closing questions, I want to ask one more question about sort of like broad lessons from doing this. Like you've been doing this for at this point, what, a year and a half more? 2 years?

A year and a half.

Yeah.

So, in that timeframe, I'm sure you'd have some really fun and occasionally sad stories and lessons. And again, sort of like, think about like top 3 things you wish people would know, or it's kind of freeform. What do you want the world to know?

Yeah. So, as you said, a year and a half, a bit more. We started with that, and I was first doing offensive— sorry, defensive solutions like firewalls. And those sorts of stuff. And then I learned you first need to showcase the risk. And then we went full red team, obviously, offensive security. But what I learned in my experience so far, whichever customer, client, enterprise you approach doing, for example, red teaming, you always need to know how to do the blue teaming. Because every project, every POC, whatever you do, in the middle of it, they ask you, okay, Great, huge risk. I want to fix it now, right?

How?

The human brain immediately thinks about fixing.

Yes.

So exactly. That's why we are now going into purple team partnership and the other things because we need it instantly. It's not I do the risk assessment and then half a year after I fix it. No, the human brain wants to fix it now. This is one learning from my side that I saw. The second thing was how much system prompt can change during development. We had customers that changed more than 500 times their system prompt within a month on the productive chatbot. And why? Because everything that was publicly exposed as an issue, they just hardcoded it in there. Like, don't mention this politician. Don't do this.

I like that.

They just tried to patch everything because the cost of rolling back the chatbot or putting it down, it was too huge. And they didn't want to do it. It would be embarrassing. So they were every day multiple times fixing the system prompt. And then what happens at the end, people won't understand that. At the end, you have so huge system prompt which you send with small user message to the LLM. So you send like 20 tokens from users and a few thousand tokens of system prompt, which is more or less guardrails for you. Yeah. So just, this is not the way you should do protection. It's better to have guardrails and not to have very expensive system prompts, right? And this was one funny mistake, like we noticed people not taking care about. Yeah.

Yeah.

No, that's a fascinating one. I think I almost, you can almost guess somebody would do that, but I didn't realize it's common. But then again, it's intuitively what makes sense to the team. Like, hey, I'm just going to load it in the prompt and tell the machine to not do it.

Yeah. Hmm. Yeah. Come on. But imagine on the other side, you're having the management calling you. Yeah, I'm talking bad about some politician, you need to fix it as soon as possible. You don't have more options than putting in system prompts, right?

Yes. Yeah. Well, you're not going to build a new product. Yeah.

System prompts. On that same day, definitely not. Yeah.

Perfect. So, we are at the end. So, I want to ask our traditional closing questions. Basically, one tip on this area for people and one resource, reading material. When we say resource, people recommend their own blogs. When we say reading, people recommend fiction books. So it's pretty much anything you want that can be read by a human or an LLM.

Yeah, yeah, definitely. So the tip for this area, I would say just go out there to any CTF or even multiplayer competitions, which I like, and try to win it because doing things that you don't know is the only way to learn. So that's much better than like reading any book or or any advices, but if I would need to like point some source where you could educate yourself, I would not even recommend the book for AI red teaming because I think more, let's say, up-to-date things are on some top Discord channels like AI Village, things like that.

Yeah.

Just get in there, like track it every day. There are really some up-to-date information which also we as a Offensive security use daily to top our attack database. So these are like 2 basic suggestions. Yeah.

No, perfect. This is actually quite useful. And I think that the concept that the Discord channels would help you more than a book makes total sense in the rapidly changing field. And that's where the wisdom is, not in the book written in 2019 and updated in 2022, because much of it would be like way out of date.

Yeah, definitely.

Perfect. Thank you very much for being on the show. This was insightful, and I think lots of lessons to various teams, defenders, and testers.

Thanks a lot, Anton. It was my pleasure.

And now we are at time. Thank you very much for listening, and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Bye.

Security. Security.