EP135 AI and Security: The Good, the Bad, and the Magical

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button in your podcasting app of choice. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on our LinkedIn page. Anton, we have returning guest Phil Venables today talking about, well, I guess it's a returning topic as well, AI and security.

Correct. But you would see, as expected with this guest, you would have a lot of surprises, you have a lot of deep thoughts, you'd have practical things. It's just a really good episode because it's a really good guest.

Is he your skip level? I think he might be.

Can we avoid this topic for now and just proceed with the show?

Yes, listeners, why don't we get right to it? Let's turn things over to today's guest. And with that, listeners, I'm delighted to welcome back to the podcast Phil Venables, CISO here at Google Cloud. Phil, thank you so much for joining us once again on the show. We are talking about, I think, a fun topic. We're talking about AI for security. And I want to start with, this seems like kind of a softball question, but why is AI such a game changer for security? And to question my own question, can we even have game changers in cyber at this point? Does what I'm saying make sense?

Yeah. Well, so look, first of all, it's great to be back. I'm still an avid listener of the podcast, so I always feel it's still a privilege to come back on. You know, it's an interesting question because everybody talks about, is AI going to be a game changer for security? And I think most of us would look back and say it already has been a game changer for security. When you look at perhaps not generative AI yet, but more traditional types of machine learning across everything we do on malware filtering, spam filtering, safe browsing, you know, you had Panos on a few episodes ago talking about safe browsing. That's powered by machine learning. Some of, Tim, your own products have got a lot of machine learning under the hood. Everything we've done with IAM Recommenders has got machine learning under the hood. So it's already been transformational, and I think we've gotta remember that. But I think certainly what has started to happen and what is coming with large language models and other transformer-based generative AI is going to be significant in the future. I think we probably won't feel like it's game-changing because certainly what we're starting to see already, whether it's use on software security, on configuration recommendation, on malware analysis, on suggesting detections based on threat information, it all feels kind of very incremental. And it's great incremental steps, but I think it's happening so quickly and so incrementally that it won't feel like it's game-changing in the moment, but we'll look back in 12 months, in 24 months, in 36 months and think, like, holy cow, this really was game-changing. So, it is incremental, but it's going to be game-changing over time even though it doesn't feel like quantum leaps every day. It's just incremental, which is great.

I think that makes sense. Like, a rapid quick step incremental in a year is game-changing. That actually, that logic makes perfect sense to me as well. Let me ask you a bit of a strange sub-question to this. We do see sometimes other vendors assume that generative AI would generate stuff and they would just run the code. And when I hear this, I'm usually a little freaked out inside. So as far as use cases where this type of rapid incremental change leading to game changing may happen, what are you thinking? Detecting threats, reducing toil, where is your mind?

Both.

Yeah, I mean, it's a bunch of stuff. I mean, certainly I'm excited by generative AI in its ability to enhance the productivity by reducing the toil of everything from people doing security analysis roles, coming up with new operations, detections, many other things like that. I think what is going to be really interesting though is, and you kind of touched on this, Anton, is the notion of how do we use this as part of secure software development or establishing secure configurations? And I think that's going to be pretty interesting. And while I wouldn't say any of this technology yet can generate software for all purposes that has no vulnerabilities in it, I think we're a ways away from that. But when you look at very specific use cases around generating framework code, boilerplate code, generating other types of code, I think it does radically transform developer productivity. And you can imagine the use case of it, you know, imagine if you're in an IDE with a generative AI coding assistant, and it watches you code up some SQL query and then says, look, don't do that. I'm gonna give you my automated framework for doing safe SQL injection-proof queries to the database. Or, I see you typing up this web server call, and I'm now gonna give you the automatic boilerplate to make that a secure call with all of the cross-site scripting mitigations in and all of the cross-site request forgery mitigations. That calling up predefined boilerplate within the enterprise context is going to be transformational for security. I mean, that's clearly possible today with many IDEs, but doing it in a generative way that has a much deeper insight into the context of the code is going to be pretty significant. And then overlaying on top of that, the whole controls as code in infrastructure or cloud configuration to be able to generate, and we're doing this, as you both know, with our Duet system, taking our AI and applying that to configuration generation. Imagine doing that, and so you're auto-generating secure configuration as part of systems development. It'll improve security as well as transforming productivity.

That makes sense. And I think to me, this is much of the excitement is there. And I think that occasionally I do have a fear that the gen AI would go kind of the opposite way, namely that it would give you something it learned from bad programmers somewhere, and instead of secure, framework, it would give you the very opposite. So I want to switch it briefly to fears from excitement. I mean, you are a CISO, and the classic, almost like a mainstream press question to CISOs is, do you lose sleep at night? So is there any AI security issue where you kind of lose sleep at night?

I think not in the way people might expect. So I think there's a lot of great work going on here at Google and in many organizations to take the risks you might expect with AI around training data, making sure that you're putting the right data in that will yield the right outputs, managing the tests, managing the security of the model weights and all the other parameters. So there's lots of challenges there, but there's also lots and lots of work going on, including our own secure AI framework that prescribes a bunch of controls for that. I think the things I worry about is when you think about coupling AI to critical systems that then actuate physical activity or cause transactions to occur or cause, you know, some type of activity to occur, and then thinking about whether organizations are building in the right input and output guards around those processes, if you like, circuit breakers to make sure the AI is doing the right thing. I think many organizations have the, if you like, the DNA of how to think about that problem of putting AI in the context of a surrounding set of controls or guards. I worry sometimes that there's some organizations that have not come up in that, almost that safety engineering culture, that won't necessarily think to do that. And then the other thing I worry about is the unintended consequences of the use of AI, because AI in many business processes is going to transform those in a way that may create unintended risks that have not been foreseen. And so again, it's definitely encouraging when organizations are deploying AI that they're not just involving their security team, they're thinking about the risk, the trust and safety, the compliance obligations. And that's definitely a, more than anything I've seen before, a whole of risk management approach to get this right, not just a whole-of-security-team approach.

That's really encouraging that you think this is going better than other things you've seen. On that sort of thread, I want to pick at that a little bit. Do you think AI is going to help the good guys more or the bad guys more? And if the good guys have security and privacy and risk and safety engineering all in the room, might the bad guys be able to move faster and do more bad stuff faster than we can do good stuff? What's the right way to think about this?

Well, I'll give a predictably non-committal answer and say it depends. So it depends on the attacker, it depends on the defender. I'm actually optimistic that all things being equal, AI should give defenders an advantage because AI is good at amplifying capability based on the data and the configuration the organization has that that AI can apply to, which in the general case, Attackers don't have as much visibility into that. Of course, attackers at certain stages will have visibility into that. But when you think about the possibilities for an organization to use AI to manage its own territory, that's inherently a defender's advantage. And again, when you think about the classic trope that I don't actually agree with, this trope of defenders have to be right all the time and only attackers once, I actually think I prefer the alternate framing of that, which is Attackers have to evade detection all the time, but good defenders only have to spot and respond to the attacker making a misstep once. And I think AI, especially for that, tips in the favor of the defenders, but defenders have got to make good use of it.

I think the one mistake, this has been always my motto for hunting efforts, right? What you just said with attacker making one mistake and a good blue team detects them and the breach is discovered or prevented. So that to me makes sense. And I think that AI superpower for that is a really good idea. And I think it's almost like it's worth the price for the entire podcast is that line. Because to me, this actually makes sense. And it's also very rarely represented in other communication. People don't really focus there. Like, that's a way where AI can give defenders superpower to turn this from the best teams can get it to like many teams can get it.

Yeah, no, I think that's right. And then you look where people are paying a lot of attention to how attackers may use AI, for example, in constructing new and more sophisticated phishing attacks and other types of social engineering attacks that will thwart authentication systems. You know, my answer to that is people should probably be upgrading their authentication systems to phishing-resistant cryptographic token-based technology such that we're putting in defenses against all forms of phishing, not some superpowered AI phishing. So I think we don't necessarily need to think of how to use AI to counter an attacker's use of AI if there's a better control that takes the game to a different place anyway.

That makes sense.

I really like that answer. People get so wrapped up about AI versus AI instead of solving the problem where AI is just the tool that's used to accomplish it.

That's right.

Yeah.

And similarly, again, if you think about attackers using AI to probe your configuration, to look for weaknesses, to identify vulnerabilities, yeah, of course they could do that. But also as defenders, we can do that. And like we talked about before, the defenders applying that to superpower their identification of the most critical vulnerabilities to focus resources on resolving those vulnerabilities before the attacker sees them. It's part of the things we've talked about before, which is the goal of defenders is to have your OODA loop go a lot faster than the attackers. And I think, again, I'm optimistic on this, that AI powers that part of the defender more than it would power that part of the attacker.

So let me probe very quickly at one argument. One part of the logical argument is that when we were presenting the— preparing the question, I was nervous that attackers would have an inherent advantage in the adoption speed or velocity, because they're less constrained by the rules. So I almost feel like I have to ask you, do you have a good counterargument to that? Because the fear is that attackers would just go faster because they have less rules to follow, them being, you know, the bad guys.

Well, it's interesting. I mean, I think defenders in defending themselves don't have as many rules to follow as people might think. Now, there would be some significant constraints. So if a defender is putting in place an AI system that automatically finds attacks and then responds by crafting defenses, and then that autonomically then adjusts defenses, and the AI is in charge of your entire security, then if that's your goal, then you probably do want to move slowly and make sure that you're actually doing that in a safe way. And clearly, the attackers don't have to be measured in how they do this unless they're really wanting to be careful about whether they're detected or not. But I think when you look at all things being equal, if the attacker can move quickly because they're not constrained by rules, they are constrained by the absence of the data they might need about your environment or the environment they're attacking. And they're constrained about being careful so that they're not detected in unexpected ways.

Stealth. Yes. Stealth means they're slow. Yes.

Yeah, exactly. Then on the flip side, as a defender, You may have some constraints because you're operating on the side of safety and managing your change risk, but at the same time, you've got, or you could have, perfect knowledge of your environment. You've got access to all of your data, and you've got the ability to change your environment in the favor of the defender.

I think there's another interesting asymmetry that gets highlighted by the AML model that Google recently released, where defenders can work together, but adversaries are constrained in their ability to work together. And I think given that AI systems require data and large amounts of data for training, this is an area where if we can get good at cooperative data sharing without releasing our secrets to each other, I think we actually have a really interesting ecosystem opportunity to be asymmetrically advantaged in training ourselves, since we're all on the same side at the end of the day, unlike adversaries who are competing to defraud companies for the same limited pool of dollars.

Yeah, I think that's absolutely right. I mean, I think, as you know, adversaries do share data with each other, but it's usually in the context of a kind of a a black market supply chain rather than the well-intentioned sharing of data for common defensive purposes. You're absolutely right. In a world where the AI is powered by broader and broader knowledge of what's going on, the more that we can share information in appropriate ways, the more we're going to be able to power that collective defense.

Yeah. So I want to shift gears a little bit and talk about defending AI systems and securing our AI Lifecycle, maybe. Does that look like defending any other app lifecycle? Is it different? Is it just a big data problem? What does defending AI look like?

Well, it's starting to interest me a lot because it's a combination of things. So at one level, it's like software security. So you have a whole bunch of elements of your AI system that you have to manage their provenance, you have to manage their secure build, you have to manage the lifecycle, You have to manage the testing and the regression testing. But at another level, it's like data security and data governance, because you have to manage the training data, the test data, the weights and the parameters. You have to show that there's provenance of that data. You've got to worry about the intellectual property. You've got to worry about all sorts of aspects of the data that is then used by the software to implement the AI system. You've got to think about input and output management in terms of the prompts and the output. And then, like we talked about before, you've got to think about the guards and the circuit breakers and the API gateways and all of the other things that surround the AI system that act as constraints on how it is used and how it can be prompted to not reveal sensitive information, to not do any outputs that you don't expect, or not create— actions in transaction and other systems that you may not want or expect. And that's a really interesting combination of controls and risks. And it's funny, the environment it always brings to mind for me is in financial services where you have security and risk management of algorithmic trading systems. You know, if you think about these high-speed trading systems, they're models, in some case machine learning-based models. They have input/output testing software, data controls, and then they have a bunch of guards and circuit breakers around their behavior to make sure they're operating within pre-specified parameters. And so, you got to imagine, I was thinking this the other day, you got to imagine that the security and risk teams in large banks, trading houses, or brokerages should get to grips with this set of risks in somewhat pretty intuitive ways, whereas other organizations may have to kind of come up the learning curve of how to arrange together the software security, the data governance and control, and the surrounding environmental guards for the end-to-end risk management of AI systems.

That actually makes sense. But you also pointed out some other similarities. I certainly see similarities to early cloud years where, for example, people didn't really get the whole shared responsibility. I mean, admittedly, I still see people out on the street who don't get shared responsibility for cloud even today. But for AI, it seems like we host the system, somebody else tunes the model, we give the foundational model. There are a lot of parties playing here, and it feels like the shared responsibility model of some sort will be in play for securing AI. So we, of course, pioneered the shared fate model at Google. So do you have any view or any kind of vision or anything on how shared fate would apply to AI?

Yeah, I mean, I think it's somewhat similar to the cloud as we think about layering shared fate on shared responsibility in the sense that As you pointed out, we provide a lot of the models, the model execution environment, many of the testing tools. And so we invest a lot in our foundation models that we provide as part of the Vertex AI model garden environment. And then we also provide a bunch of end-to-end data governance tools, surrounding tools that control the environment. So we're investing a lot of our opinionated knowledge of how to run these models that we use in other parts of Google into those products. So, very, very similar to what we do with many of the cloud products and layering in that opinionated guidance and taking responsibility for how we've done that is a similar moving up the stack that we've seen in other applications of the shared fate philosophy.

And that means that we would have the similar kind of mechanisms for helping clients with what's on their end. of the shared responsibility model. So, like, pre-secured elements to assemble, landing zones, basically, some of the ideas from how we manifest shared fate on the cloud would apply to AI systems.

Yeah, exactly. And then how you think about taking the foundation models, helping customers develop their data governance practices to bring their training data inside their secured customer project where the Training data that develops the adapter weights that sit on top of the foundation model don't make it back into the Google environment. So customers get to protect that, but they then get the apparatus within the Vertex system to be able to manage that. Again, it's giving people the tools to run by default in a controlled way as they bring their data to then use the foundation models to deliver a specific use case for them.

That makes a lot of sense. And it pains me as always to do this, Phil, but I am at the point where I have to ask our traditional closing questions. One, do you have recommended reading, maybe specifically in the area of AI security? And then 2, a recommended tip for organizations to want to start pulling AI into how they defend themselves?

Yeah. So like the reading is, you know, I'll be a little bit parochial, is that, you know, there's so much good Google content either within Google Research And I'm sure you guys can add to the show notes the link to the research website. But then we've done a lot of really good documentation. Many of you will have seen the Secure AI Framework that we published. And then we also published recently a Google-wide red team guide on how to think about red teaming AI models. A big part of the testing of these models is not just to do the normal data-driven testing, but do adversarial testing against not just the security, but their overall behavior. And we produced a really good red team guide That's worth looking at. The other thing I would say as well is, you know, in terms of, and I've kind of touched on this at a few points in the conversation, the important thing about AI security is security is not just the only risk attribute to pay attention to. There's many, many different risks here, whether it's everything from intellectual property to trust and safety and compliance and behavior, as well as all the operating risks of, how an AI is plugged into your wider business or mission process. And I think one of the key success factors for many organizations is not just how good a job the security team does, but how good a job all of the teams that manage all of those risks partner together to deliver an integrated risk framework for that organization. And then also, I mean, we've talked a number of occasions about the role of the board of directors. For many major companies, that's also involving the board because I'm finding the board is Also very, very conscious and highly thoughtful about how to think about the responsible deployment of AI. And that's something that we've partnered with many customers on as they build their whole risk framework. So again, not just the security aspects, but how the security team works with all of the risk functions to deliver safe and responsible use of AI while driving innovation forward as well.

That's such an interesting answer. And I wish we'd had more time to actually talk about the board piece because I think there's so much interesting stuff happening at that level too. It really, I don't know, sometimes I feel like security can be a bit of an afterthought for people. And this is one place where the intersection of AI and security and board governance are just, they're really coming together right now.

Well, it's funny because, like I said, we've been spending a lot of time with boards, and I find it really refreshing that this is a topic, as security overall increasingly is, but it's a topic that boards are considering as they think about how their companies are going to innovate with AI, I've yet to come across a board that isn't also very, very focused on that trustworthiness, safety, responsible AI. And it's been particularly pleasing, you know, for all of us at Google, where I think we more than pretty much anybody through the development of this technology have been very, very focused on that responsible and bold use, but really thinking about trust and safety and security and compliance and privacy. And all of those risk mitigants is a core of what we build. To have customers, boards call that out and thank us for being focused on those topics because they are focused on it, even amidst the pressure to innovate, is actually a big positive for all the people at Google that work on these things.

And listeners, Phil's not just saying that. I mean, you go use the restroom at Google and there are signs in the bathroom teaching you about AI ethics and compliance. It's really everywhere here, listeners. And I think— I think it's easy to say, oh, Phil's just saying it, but really, we really do care about these things. And so maybe on that note of Google really does care about this stuff, and it's great to see everybody else caring about it, I want to thank you, Phil, for joining us once again.

Yeah, always a pleasure. Thanks, guys.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security! Security!