EP283: How Google Cloud CISO Chris Betz Uses LLMs to Defend Billions of Users from Vulnerablities

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here today are myself, Tim Peacock, one of the group PMs on Google SecOps, and Spencer Lichtenstein, the group PM leading SecOps' efforts in agentic analyst experience. And God, how many other things, Spencer?

Yeah, a couple other things.

Delighted to have you join us as a co-host today. Listeners, you can get this podcast wherever you get your podcasts, as well as at our website, youtube.com/@cloudsecpodcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. Spencer, what, who did we have today and what did we talk about?

We had Chris Betts, the cloud CISO here at Google.

Yeah. Pretty cool.

Yeah, it was, it was awesome. Super interesting insights on, I mean, basically to me, all the ways that they're trying to use kind of AI to shift defensive posture. you know, closer to codebase, find, fix, you know, patch kind of things earlier in the cycle. Yeah, great conversation.

You got an effect named after yourself today. That's pretty cool.

Yeah, that was fun.

That is fun. Listeners, you'll have to stick around to find out what that is. Maybe without any further ado, let's turn things over to today's guest. Today we are joined by Chris Betts, CISO for Google Cloud. Chris, thank you so much for joining us today. We're talking about Google's new AI Threat Defense Framework. And when talking about it in the phases, which we'll talk about in depth, one of the things that really stood out to me was during the prepare phase, you were talking about not just allocating headcount to vulnerability programs, but also token budgets. And so in contrast to a traditional world where we think about headcount time, how is this going to work for CISOs going forward? How should people start preparing for a token budget world?

Well, first of all, thanks for having me. It's great to be here and it's great to be able to have this conversation. So appreciate you making time to connect with me on this. That's a really interesting question because, you know, as CISOs, we are all responsible not just as technology leaders, but as business leaders, as leaders that understand the business, understand how to use resources in order to help the business move forward. But I would certainly say that I don't think of tokens as a headcount metric. Token budget and compute budget, I consider part of the financial and resourcing side of the business. You know, AI compute and resources exist to give leverage to our people, to allow them to scale and do more. And so these are the resources we provide to the teams to enable them to be successful. So while we certainly need to manage it, that's not to say that this is about, you know, counting it like headcount. This is about how do we equip people with the tools they need? Or as we like to talk about, how do we let smart people solve hard problems and computers do the rest?

Hmm.

And so this is down that line of computers do the rest. This is just like we buy software, you know, back in the day for me it was IDA Pro. Now it's about tokens and compute resources to make sure that we're equipping people to do that well. And of course, operationally, this looks similar to other resources we're gonna have to manage, and we do already manage. We get allocations, we ensure they're program-aligned, they're giving us the max impact. We're looking at where they're spent, what return we're getting for it, because our goal is to maximize the impact and really leverage the people so that they can solve the hard problems and have the right tools to solve them.

So we're impact maxing, not token maxing.

Exactly.

Yeah, nice.

So in, in the second kind of section of this framework, you talked a bit about scanning and prioritization. Especially kind of regarding things like code. And one interesting thing I saw was you highlighted that the test harness and domain expertise, from your opinion, were more valuable than sort of a core frontier model. And yeah, especially when weighing kind of those 3 factors. And just curious if you can talk a little more about why that is, kind of compared to raw model horsepower, why the human expertise and that harness are maybe a more important mechanism for the scanning and triage phase, especially when I think for many years we've talked about human expertise maybe being replaced by kind of tools. You know, how are you thinking about those being maybe a more important mechanism than the model itself? We can start there.

Oh, that's an important point to bring forward because at first glance it may be counterintuitive, but for all of us who have ever, you know, used AI to help write code or debug a situation or anything else, we all know how important it is to really give the model the right work to do to help focus it. And in fact, finding vulnerabilities is no different than other areas in that it is really complex. You know, you've got memory, you've got the context, you've got what gets attention, where you focus it, how it approaches the problem. Don't get me wrong, you know, when you've got modern models that are kind of mixtures of experts and have expertise, That expertise comes through. A model with better expertise certainly performs better than one that doesn't. But how you bring the data to the model, how you focus that model, how you give it context, memory, and other aspects are so critically important. And so a human researcher who knows how to do this function can help the model be 10 times more effective. 10 times, of course, being notional. Similarly, the harness that helps drive what content is available to the model, how it focuses, how it works, is so incredibly important. So that combination, the harness being developed and led by the user, is really, really impactful. If you asked me, you know, and our data supports this, we have almost all of the vulnerabilities found by some of the most expensive models can also be found and are also found by the less expensive models when driven by the right harness. Now, the best combination is, of course, when you get all 3 together. You get the best model with the best harness with the best researcher. That's when you get the maximum impact. But it, it makes such a big difference. The one other thing I'd point out here is that one of the key factors— back to, you know, the first question about token budgets is cost. And so knowing when to use a more expensive, more complex model, and when to use a model that may be a generation behind but is completely sufficient for that first pass with a really good harness, is an important thing that we're all going to have to learn as we go forward so that we're using that compute, we're using those tokens in the place for maximum value.

So it wouldn't be an episode of our podcast without, you know, somebody asking our guests to make a prediction that Nobody is qualified to make. So I'm going to ask you to make a prediction, and I want to draw an analogy here. In the world of super competitive chess, centaur teams, human plus computer hybrid teams, were dominant for about 15 years before things like our Alpha Chess here from DeepMind really swept the field. And so we're talking about this human harness model triumvirate. Do you think that's going to last, or do you think we're looking at a world in the future where it's just going to be the best model always finds the most vulnerabilities and the human expertise eventually gets left behind, shall we say.

In chess, there's kind of an ultimate stop, right? Because it's an artificial game. You can stop, you go up the layer, you start off with human only, but eventually there's a stopping point. You don't get anything more meta than a game of chess, right? It's just the game of chess. When I think about finding vulnerabilities and vulnerability research, I still think that there's more layers above. Like, Finding vulnerabilities in a set of code. Today, you take a library, you take a component, it's that triumvirate. It's a combination of the human, the harness, the piece of code— sorry, the model to find the piece of code. As things go on, like maybe that model gets really, really good at finding a vulnerability in a piece of code, but now of course you've got linking multiple vulnerabilities together to lead to a risk.

Right.

And again, models are good. The best combination of a human, a harness, and a model today is much more powerful than a model alone. And the best models are better than less good models. Again, now you can start getting— but I think that these layers in terms of looking at systems, looking at systems of systems, not just from a technology standpoint, but also how vulnerabilities are exposed, I think there's gonna be a long period of time where what we're talking about is humans increasingly giving more of the work to computers, computers solve the rest, and the people being able to take on more uniquely those smart hard problems. Ultimately, like, we'll have to see where this goes. I can't predict that far out.

Mm-hmm.

But I do think there's much more room for growth in this space as we start moving up layers and we start thinking about orchestrating and coordinating. I'm also gonna flip this around though, man, The day that we can ship code and have high confidence that it has few or any bugs, security vulnerabilities or other bugs, that's going to be an incredible day. Can you imagine what we're going to be able to do with technology in that day where we're not chasing bugs and we're not chasing vulnerabilities and instead we're bringing brilliant features forward? What an amazing day.

The cynic in me says that'll be exactly like today and there will just be fewer of us sweating in the back about cybersecurity. But, you know, we'll, we'll see how that shakes out. I do like the point, though, that bugs are just another class of defect, and that's a theme we've had on a lot of episodes of the show over the years, listeners, that security vulnerabilities are just another kind of defect, and the techniques that we use for normal defect discovery and handling apply to security as well.

So, moving into another piece of the framework, which is some of the remediation phase, there's this discussion of the blast radius to make sure that as something is fixed or changed, you don't kind of break everything, right? And you talk about kind of flipping that around and talking from the foundational code level of, like, having that be maybe one of the biggest blast radiuses. And instead of sort of going back and maybe fixing something, like, I guess, maybe to use an old term, like, doing this shift-left thing, maybe in a more extreme way. I think the biggest thing, like, that occurs to me is how do you convince people to let you test things that early in their development pipeline, right? Like that early in what they may view as like slowing down their business, slowing down their process, right? Like if you believe in AI defense and shifting it all the way back, what's the key to getting people to buy into that? Or have you seen any success stories of like how you get these teams to believe in the standard engineering wisdom versus, like, this new age of engineering wisdom of shifting things so far earlier in the process and letting AI potentially try to, you know, remediate kind of things before they ever get into the codebase to begin with?

I like the question. One of the things, one of the biggest things that's behind where we're headed, behind the decision to go and invest in these areas first, is what we need. We know that with AI, other folks are going to find vulnerabilities, and we're going to need to patch them and patch them quickly. And so there's 2 underpinning thoughts here. One is we need to have the confidence that we can roll out changes at scale when a vulnerability is found at scale. And so, One way or another, we need to solve this problem. We have to be able to solve this problem. Otherwise, we're not going to be able to respond when we don't have a choice. So when I have a choice, when I'm going and finding my vulnerabilities, that's the time to learn how to do it. The second thing is we've got to focus on what's most critical. If we spend our time on the small problems, if that's where we're investing first, We're staying a lot, very exposed for a long period of time, and we can't afford that. And so those 2 things said, hey, the things that we should focus on first are the ones that have the largest blast radius, and we need to work on figuring out how to do this best. Now, that doesn't mean that we just go, all right, hey, you only live once. We're just going to roll this out and see what happens. It means that we have to develop smart ways to do it. That doesn't mean zero blast radius. It means we have to be able to roll out one section at a time. We have to focus on how do we make drop-in ready to the greatest extent possible patches? How do we make it so that from a functional standpoint, the updates are as similar as possible? Because when you start making functional changes, that's where you often have the regressions or drive other changes. Other code has to change, right? Because— The thing that you're changing, let's say boring SSL, doesn't operate the same, that would be a problem. That would require hundreds and hundreds or thousands and thousands of changes. And so, how do we make things that are drop-in ready? How do we make them small? How do we do the testing? How do we gain confidence? What's our rollout plan? And solving for that now, when it's a situation that's under my control, is absolutely critical. And solving any problems in that code now. when I have a situation under my control is, again, absolutely critical. And so, this is about focusing from a reliability standpoint and a security standpoint on the most critical capabilities and working on those first.

I never thought I would have the CISO of Google Cloud invoke the YOLO principle on the podcast. So, I, I really appreciate that answer. And I really agree that these moments in security where everything seems to change all at once, also point to nothing has actually changed, where it's really the fundamentals and getting the fundamentals right that will keep your organization safe at the end of the day. Because whether we discover the vulnerability through AI and whether we develop the exploit code 3 times as fast, if you can't patch reliably and can't have confidence in your process to do that, you're not going to win either way.

Exactly. You've got a nice science experiment. You've done a great job as a security team, but you've actually failed to protect your customers.

I think an interesting— I'll just interject and throw you off for a second. One One interesting observation of just what we discussed is, like, patching, right, is, like, one of the oldest problems, I feel like, in my 15-plus years in security, in this collective room's 125-plus years in security.

I guess that leaves Tim as the old man out, by the way.

I just wanna point it out. I'm glad you picked that up. We're talking about accelerating, identifying vulnerabilities, fixing code, et cetera. But this patch problem is still a very hard thing to get right and not impact the broader organization in a disruptive way. We have yet to see, to me, a truly innovative kind of AI-driven patching mechanism come up that maybe does what you're describing, Chris, of identifying a specific area that's critical, but has less blast impact if it goes wrong, and deploying the patch very targeted there to accomplish what you're talking about. And so, it's just interesting, like, some of the oldest problems are still unsolved, even in the era of kind of having shifted a lot of this code development way to the left and being able to do it a lot faster. Maybe someone will.

I think that's a great point. This is a hard problem. Just because a problem's been around for a while doesn't mean it's easy to solve. Like, the reason this has been around for a while is that it is a truly hard problem. I think one of the things that I'm very interested in, in fact, we've been talking a bunch about, is it is almost impossible to change code without causing some sort of side effect. That code that executes may be faster, it may be slower, but how do you both minimize, and then how do you accurately describe the side effects of the change? is so important. You know, one of the things, there's a good XKCD cartoon about how one person's bug is another person's feature. I believe it involves holding down a spacebar to overheat a CPU.

Yes.

I appreciate that. It's something we've all run into in our engineering experience. And, you know, understanding and being able to describe what the change is and what the impact is, the more we're able to do that, I think that could lead us in that direction, Spencer, of, What are the changes? How do we minimize them? And then how do we describe them so that you can make a judgment about, is this meaningful to the code or is this drop-in ready? And I've seen some small experiments that are actually looking very, very promising in this space. There's more to do here, but I do think that kind of the combination of the security work we're doing, and some of which we're talking about here, plus some of the advanced engineering work, that I see going on around Google and with Google DeepMind. Like, that combination is taking us some really interesting places very fast.

Spencer, I think what you described is the security equivalent of the Lindy effect, which says the longer a non-perishable work or idea has been around, the longer it is likely to persist and stick around. I had a conversation with about a dozen CISOs in Vegas during Next, and I, you know, I asked the room, you know, raise your hand if you think you're going to patch faster as a result of AI vulnerability discovery, and not a single one raised their hands. So patching, I agree, remains a hard problem. Chris, back to the paper that you put out, you talk about the open source strategy of refresh, remove, and rewrite, which is a nice triplicate of things that start with R. It sounds resource intensive to rewrite things. And so how should CISOs out there think about their resource allocation here? against those different options for, you know, core dependent on but potentially insecure components?

I look at both remove and rewrite as adjacent solutions, for what it's worth. As we were doing some internal looks, we found that 50% of the open source we consumed had fewer than 500 lines of code.

Okay.

So when I think about remove, I think, does it make sense to take a dependency on an external package, or does it actually make sense for us to just remove that dependency and put in the right situation? I'll tell you, 5 years ago I would've said, oh no, the best way is to use an open source package. Now today, hmm, that's a really interesting conversation because— Using AI-generated code, how long does it take you to get 500 lines of code? 500 lines of good, well-tested code is still very achievable in a reasonable amount of time. And if that doesn't require you to do the very hard act of making a patch, there's some value there. Rewrite, of course, is, all right, we're going to need to maintain this library, but we all know that the massive majority of vulnerabilities found by AI models these days are memory safety issues. And so taking some of the packages written in non-memory-safe language and porting them to memory-safe languages can be incredibly powerful. Now, Tim, to your point, that's expensive. That's where honestly we're doing a bunch of research right now because I want that to be AI-powered. You still have to have an incredible harness behind it. In fact, that's where teams are spending a ton of time. You have to have some incredible engineers overseeing those, but making it possible to rewrite a complex, large open-source package over the course of a week all of a sudden becomes both achievable and a worthwhile investment, especially against the cost of patching. And so really, It's all of the pain that Spencer was talking about in terms of patching that we have to put this up against. And so if we believe that we're going to have to do multiple rounds of patching on the same library, we can even call it the Spencer effect, by the way, due to the vulnerabilities, then the Spencer effect takes hold and we go, actually, if we can get AI writing in place, then getting the AI writing is actually the right solution. And that's where we're investing today. And by no means have we solved it consistently at scale, but that's exactly where some of the smartest people I know are spending a ton of their time, and we're making good headway.

I love that we've managed to coin a new effect on the show today. I think that's actually a first in our 280 episodes to name something after a co-host. So congratulations, Spencer.

Wow, this is incredible. I would say I'd love to co-host again, but I'm not sure that there's anything to beat this now.

Yeah, no, you should just go out One and done. This is great.

Yeah, yeah. Chris, you actually bring up kind of one, one last piece I'd love to ask about before Tim closes out. You talk a little about all of the, the stuff that kind of we're doing at Google, right? And look, it's a different place, right? Google's kind of a unique company, as you know. You're newer here. I'm sure you've seen like we operate a little differently than some organizations. We obviously have some really unique abilities that we're very grateful to have as far as just DeepMind and Cloud and all these other core pieces of Google to bring together. We're discussing AI agents writing patches, AI doing red teaming, kind of trying to make the most use of AI in all these different ways, but also making sure there's some human oversight. I think if you're an organisation from the outside listening to us, you might be like, okay, this sounds amazing. It's great if Google's trying to do this. How on earth do I try to apply this to my own organization and make sure that some human has some oversight, or kind of what we tend to call human on the loop, to view what's going on? Do you have any advice or insight for organizations listening, thinking about how they might begin adopting this blend of... Agents accelerating their defensive posture while also not feeling like they lose control of the ability to see what's going on and not be the bottleneck?

A few things to approach that. And Spencer, you're pointing at some of the hardest things because you're not just talking about technology change, and we all know that's hard. You're talking about culture change and in some cases process change as well. In fact, I think it's all of those at the same time. And so I, I think the first piece is take a step back as a security person, partner with your engineering teams, and talk about how you want to work differently, what would have to be true for you to be able to operate differently, and how you gain the confidence to make those next steps. So I don't think it's something security teams do on their own. It's a conversation with engineering teams to say, hey, here's what would have to be true. Here's how we would get there. Here's what we think the destination might be. We'll adapt over time, but let's head towards that destination. And so, it requires a change to your software development lifecycle. It requires potentially a change to your deployment practices. Back to our conversation about testing and rolling out. Like, these are the things that you need to build together. In order to be able to move forward. And the technology may— you may take one step at a time, but if you don't have a change in mind, if you haven't figured out that you want to go to a different place, you're not going to get there. The second thing is culture. And part of changing culture is leading your team and asking them to help change. And part of it is making sure that they have the right capabilities. Back to Tim's first question about token budgets. If you don't give your team the resources that they need to figure out how to do it and how to do it differently, you're not going to be able to lead that cultural transformation. And then the last piece is go find those folks that you see doing it well. One of the things that I appreciate the most about the CISO community, the security community in general, is that we all see each other, whether at the same company or not, as colleagues, because the adversary's on the outside and we're on all the same team of defending ourselves and our customers. And so go grab other people, bounce ideas back and forth, figure out what's working, start with the small wins, and then scale out from there.

That is something I absolutely love about our industry as compared to really any other. This is— at the end of the day, we're all on the same side, and I do love that about security. The thing I hate to do in security is close out episodes of my podcast, but I have to. And so, Chris, Traditional closing questions. First, do you have a tip to help CISOs prepare to lead that cultural change? And then 2, do you have recommended reading? And that recommended reading could be truly anything.

Yeah. So the tip to lead the cultural change, it's the one I just gave. It's go make an investment, go give your folks a set of tools, give them the ability to go play, experiment, and try to do their job differently. 'Cause if you don't do that, you're just not gonna— not enough folks are getting hands-on. They're stuck trying to do their same job. You've gotta give them both the time and the space to try something differently. On reading, I'm actually gonna take us away from cybersecurity. I'm gonna recommend reading Thank You for Arguing, because so much of our job as security leaders is about how you talk to other people, convince other people, and engage with other people. And if you haven't read the book, Thank You for Arguing, It is a brilliant book on rhetoric and just does a great job of both helping you understand when you're being sold, which is important, as well as understanding how to communicate more effectively with others.

I have the feeling that I'm being sold right now on this book. Is that right?

The subtext involves both Homer Simpson and Aristotle in the same sentence. If that's not a selling point, I don't know what is.

I love that. Well, Chris, we've gone from YOLO to getting the fundamentals right to Homer Simpson and Aristotle all in one episode. Thank you so much for joining us today.

Great to meet you.

Thanks for joining, Chris.

And now we are out of time. Thank you very much for listening and, of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google. Security.

Security.