EP255 Separating Hype from Hazard: The Truth About Autonomous AI Hacking

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Stuff in Google PsychOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we have a returning guest today, somebody we are both delighted to have on the show talking about, frankly, a terrifying concept.

The apocalypse. It's a lighthearted subject. It's the apocalypse. Let's just be honest about it.

The apocalypse.

The apocalypse. It's like, let's just not— why beat around this particular Apocalyptic bush. It's the apocalypse. Yeah, yeah, sorry. It's the flight part that's the subject of the hacking apocalypse.

Yes.

The end of the world as we know it.

Correct.

But I think by the end of it, not to misquote R.E.M., you and I both kind of felt fine.

I would say so, yes. It went really up from the very depth of the apocalypse to like excitement and curiosity and positiveness. So I don't know how we did it. Maybe it was the guest, maybe it was us, maybe all of us.

Could be. Listeners, it's really a shame that we were an audio-only podcast today because you missed Heather's cringey face. You missed Anton using a Franken-style switch to illustrate changes. We really are looking forward to becoming a video podcast at some point. But maybe without any other spoilers other than, you know, this is a show about the apocalypse, let's turn things over to today's highly distinguished guest. Today we have returning to the show Heather Adkins, VP for Security Engineering here at Google. Heather, it is a pleasure as always to have you. This episode kicked off when I read something that you co-authored, and I got very frightened. And that doesn't happen a lot, that I read something and get frightened. And so in this article, you've got this term of AI hacking singularity, which sounds a little bit like sci-fi, but you and people that I believe are credible use it to describe a very imminent threat. And so how much of this, between us, is realistic versus how much of it is hyperbole to shock people out of their complacency?

Yeah.

You know, how much is it something we observed? How much is it something we wanna scare people with? Are AIs really hacking things at scale already?

Well, great to be with you guys. Really honored as always. I should give some credit to Bruce Schneier and Gadi Avran, who I co-authored this article with. The AI hacking singularity as a concept is something we are all talking about quite a lot. And I don't think that anybody knows the precise answer to your question. I don't think we'll know for a couple of years. Here is what I do know.

Hmm.

Number one, we are starting to see productivity enhancements and task enhancements using large language models in the hands of threat actors. We are starting to see red teams start to develop end-to-end full kill chain tools to do penetration testing, red teaming simulations. And we are starting to see large language models used to discover vulnerabilities. Now, it's just a question of time before somebody really, really smart Starts to put all of these things together end to end. And the thing that I fear the most is that somebody will end up with a capability where you just type into the prompt, go hack your favorite company. It'll do all the research, come back in a week with a root prompt.

Mm-hmm.

That's the thing that I think we are hypothesizing will happen. Whether that feels like a catastrophic world-ending event, I think it'll be a slow ramp. over the next, say, 6 to 12 to 18 months. It may not be a surprise once we do finally see it because we will have seen all of these developments along the way. That's not all that's happening though. We are also seeing defense pick up these tools and use it for the same purpose. And so it may not feel as shocking. But I will also say, I think in 2 years when we look back in time, everything we hypothesized is probably wrong and the world probably developed in a really different way. But we should be thinking about this. It should be on everyone's mind. should be imagining a really different world.

So before we go to sort of like who would wield this weapon, I want to be briefly, maybe gently contrarian to this, is that in many cases, at least in my experience, when we predict based on fundamentals, we miss another fundamental. I don't know what it will be in this case, of course, that sort of makes the future not come. And it may be a good future or in this case, a bad future. So like your example with Here's the machine. I demand that this machine hacks, and I name a company somewhere on this planet. And in a week— and it's interesting you said in a week, because if you'd say in 5 minutes, I'd be like, nope, there are fundamentals against it. But let's say in a week, it comes and says, here's your root prompt to a system in that company. So let's run this scenario. So a lot of things you described make sense. They're like bricks in that wall. But are there bricks missing? Other situations when, for example, certain steps are destined to be so noisy that the defenders will pick it up, and so you would have your prompt for 2 minutes, or maybe there's something else. I'm kind of curious, can you argue the other way, is that even though there are good arguments for why it'll happen, here's one why it won't?

I think it's good to be a skeptic, and I think it's good to challenge it. I think the skepticism should not prevent you from thinking about the problem. But I think you're right. If we look at how large language models reason about deep research, for example, they often end up in very strange places. So you could imagine you've got an AI-powered module on an endpoint that's trying to figure out how to hack a machine, and it suddenly finds itself deep in the depths of CPU vulnerability research, when actually it could just use a well-known bug that's on the machine, but it ended up like thinking about this other thing because it wasn't constrained in any kind of way. We see that actually in some of the vulnerability research that we've been doing is that the model just ends up kind of on a strange thought path and you have to rein it in. Right. And that's true of vulnerability researchers, the human researchers, and actually hackers as well. They kind of have an intuition of when to stop pursuing a path because it's just not going to be fruitful. The LLM has no concept of, hey, this isn't useful, I'm going to change direction. So that's one thing, but that is, I think, a short-term problem. Someone will figure that out. There's also the question of where does all the GPU, TPU resourcing come from for all the inference? Again, I think someone will figure this out. We already know you can run small models on regular hardware. At some point, everybody's machine's going to have a GPU. You can just exploit the one that comes in the box you've owned. Where does the infrastructure come from? Well, pretty soon, I'm sure, Internet of Things devices that—

Oh no, your washing machine doing the AI hacking, that's worse than it's doing crypto mining, frankly.

Yeah, exactly.

I'd rather my washing machine crypto mine than AI hack.

And your oven and your refrigerator are going to be a part of this sort of, you know, massive problem, right? And so, I think these small, like, things become overcome. Now, remember, If you are a threat actor who's out for a little bit of fun and profit, I think nation states have a kind of a different problem to solve. But like, I think of some of the shiny hunter lapsus threat actors that are out there that just want hacking points. They're okay with letting some of this messiness happen, right? They're going to try anything that will make them successful. So I think we've got to keep an open mind about what's realistically able, you know, people will be able to solve some of these things.

But let me ask you then the flip back to positive. Well, I guess positive would be the attackers win, so maybe not so positive. So I'm kind of thinking of maybe before my time when people came up with the first vulnerability scanner, the whole Satan story in the 1990s, when there was a major freakout. Oh my God, you can just scan and find vulns? Holy cow. No way, right? And then there was a Metasploit moment. So Is there a moment to watch for in this case? Like, what is the Metasploit equivalent for AI-powered exploitation that we, you, me, Tim, need to watch for? Like, this is happening.

Yeah, how do we know when it's here?

I wish I knew. I think it'll be obvious when we start seeing threat intelligence reports on it. We are already seeing threat intelligence reports out of all of the major threat intel companies and teams about little pieces of the kill chain being weaponized. So the Google Threat Intelligence Group just came out with one where they map out what are these different tools doing. There is no tool that has an end-to-end, but someday they will see one. I also— I think this is hard to predict because we don't know how it'll get used. Now, if it's kind of weaponized in a ransomware toolkit and sold on the underground, you may see the rates of incidents increase. But if it's closely held by a threat actor who's got really specific targeting, we may not know, right? You may not be able to tell that it's a fully automated platform on the other end. And so, I think we're only gonna know when it's physically in someone's hand.

Right, the Metasploit moment to me isn't the APT has this machine. Like, remember we had this moment about like, what if a nation state has built a quantum computer? that does break crypto today. Like, that was our moment at the other show. And I think that here, I worry less about that. I worry more about the democratization moment. So, I feel like one APT group in a certain country has it versus /github/AISploit. Please don't use the name.

Oh, no. Do not let Anton name things, listeners. That's how we ended up with EDR.

Yeah, I think the best proxy for this will be when we see it in the open source red teaming tools.

Mm-hmm.

Hmm.

That makes sense.

And I'm surprised it's not in Metasploit already, to be fair. I bet somebody's got that.

Somebody's working on it.

Yeah.

So Anton brought up this, like, you know, cryptographically relevant quantum computer fear of collapsing crypto overnight. I wanna talk about kind of the worst case scenario you can envision on the AI-enabled hacking.

That's quite bad though, Tim. You do know that, right?

Yeah, yeah, yeah, yeah. I wanna take us to like the dark place.

That's Heather's imagination, right?

Yeah, yeah, yeah. No, Heather's imagination is perfectly scary, and I'm asking Heather Could you unleash your imagination on our listeners? What happens in the bad case? What does that look like? Is it the whole world collapses? What is the outcome here? The bad outcome.

Yeah. Well, I also have really strong opinions on the quantum computer thing, because I don't think we're going to know, and I don't think it'll be a capability held by many, and I don't think we're going to know. I think on— I will answer your question directly, probably. On crypto, we should worry about advances in math. On classical approaches, not just quantum, because my friends at DeepMind kind of look at me funny when I ask this question, but, you know, how novel will the math get?

Hmm.

As the models become more sophisticated, do I need to worry about the crypto for non-quantum reasons? But on your question of what does it look like, I think there's a few possibilities. One, somebody could unleash this as a worm and we have a Morris worm type event. And that could be accidental or on purpose.

Mm-hmm.

We wouldn't know until fully investigated. I think it's possible it could look something like Conficker, which was a worm that didn't really do anything. It just sat around and everybody found it and panicked, and we wrote 1,000-page government reports on it. But like, nobody ever activated it to do anything as far as we know. I don't know, maybe an altruistic person unleashes it on the world and it patches a bunch of bugs for everybody. That could be a possibility. I think it's really just going to depend on who puts all the pieces together and what their motives are. Those are all the worst-case scenarios that I think about, but I do kind of wake up every morning and now wonder, is today going to be the day when the whole internet responds to something that just either intentionally or accidentally got out?

Hmm.

You know, we'll be shocked by it, but we should not be surprised by it if it happens.

Yeah, I think it definitely— this one, when it happens, it's going to be filed under shocking but not surprising. That part is very like— it's probably the best use for that line, frankly.

Yes.

So, now I'm going to get to a depressing subject, and I'm going to avoid the trigger word defender dilemma because, you know, you and me had some discussions about this. And for those who are not watching the video because it's an audio podcast, Heather is making this slightly cringy smile at me. But this aside, The scenario and the stories so far and your descriptions and the facts point out at a pretty strong attacker advantage in this particular case. So if attackers are operating at that speed and if they have this quote unquote magical hacking machine that you put the company name and it just hacks it in a week, defender seems outmatched. And while we sort of kind of say that to defend against AI, you need AI, sometimes it's a cliché, sometimes it's the truth.

Yeah.

Sometimes it's probably false because occasionally an axe is the right defense. You cut the wire and that's it. You can't hack. But what is the genuinely useful tactic to counter it? Or what are some of the ideas? What is this genuine AI-to-AI counter tactic that doesn't just become fake or whatever?

Right now, our definition of winning in cyber is preventing attacks.

Right. Not losing.

Yeah.

Patch your systems, make sure your passwords are strong, use a security key. We have to change the definition of winning. The definition of winning has to not just be, did you get in or did you not? It has to be, how long were you there?

Hmm.

What did you get to do? And I think what this comes down to then is the defender's job is to change the battlefield in real time. And so, you know, this is not all that novel. We've been talking about immune systems. That was the original purpose of antivirus. Detect anomaly on the endpoint, clean it up, you know, move the files out of the directory so that it can't run anymore, reboot the machine. This is not like all that novel, but the problem is, is you're gonna have to extend it in this concept of changing the battlefield to almost everything. And just to make it practical for people to think about, if you're a, a cloud-based infrastructure and your cloud instance thinks it's compromised, It should turn itself off. If you are trying to abuse a privilege that is given to an account, maybe in an infrastructure, like a, a service account or something, and it's starting to do something anomalous, you shouldn't let it do it. Like there should be something orchestrating it that just turns the permissions off. And so we're gonna have to put these intelligent reasoning systems behind real-time decision-making and disrupt decision-making on the ground. And you don't want to do that in a way that causes reliability problems.

Hmm.

So, you have to choose the right decision. Maybe it's a look to a human to approve this, or, you know what, I'm going to shut this instance down, but I'll turn up another one over here. Or, you know, there are options other than just the on and off switch, but We're going to have to start reasoning about real-time disruption capabilities or degradation, right, to use kind of all of these in the information operations playbook that change the battlefield. And it'll confuse an AI attacker just as much as a human one. And the AI attacker may actually be even less resilient to that change because it doesn't— it's not going to be able to—

Hmm.

know that it is sort of stumbling around in the dark a little bit, right? So I think that's what it looks like. We change the definition of winning and we use all the tools in the toolbox to degrade and disrupt, and we have to put intelligence behind that.

So is it— I'm sorry, I got a really weird and very uncharacteristic thought about this. Remember, Tim, this whole drive, AI-powered drive defense against ransomware where it stops syncing? It reminded me of some kind of futuristic, ultra-sophisticated version of that when it sort of disrupts the environment enough so that expectations of the attacker are just false at this point. But it is also— decisioning is supported by AI, not just by, oh yeah, here's a big switch, let's pull it. No, that wouldn't cut it. And of course, the speed wouldn't be there. And of course, frankly, decision-making wouldn't be there either.

Yeah.

But that sounds plausible enough.

Plausible enough. That's high praise from Anton.

Come on, this is Heather. I'm a fan and all that. So no.

So I want to shift gears and put my foot into a can of worms, or maybe put my foot in my mouth. Vulnerability discovery. AI is getting good at it. How does this impact software supply chain and open source and maintainers and all of that? What's the future there look like in this AI-powered world?

Let me give you the long-term outcome I'm hoping we achieve, and then we can talk about the messy middle and the roadmap to get there. Long-term, the hope is that we have AI assistance early in the SDLC available to open source developers, commercial developers, students, hobbyists, people who vibe code, that catch and prevent the large majority of classes of vulnerabilities from ever reaching a production environment. That's the outcome we want. So, if you're coding and you try to create an integer overflow, it's going to tell you, you know what, that's a bad idea. Let's not do that. Let's not ship that code. That will come. The models are really, really good. They're improving all the time. The cost for inference will come down. We'll get strategies to make it efficient. I believe all of that is possible. The science is, is, you know, we can see the science.

Mm-hmm.

The path to get there is gonna be a lot of trial and error and a lot of new norm setting.

Mm-hmm.

And I think it's for a commercial development house. So if software is the business of your company, this will be an easier thing for you to adopt. But I think for open source developers, where it's really about volunteers and it's a creative process, in many cases they see themselves as creators and a community. And we built this thing not just because we like the software, but also because we like working this way. They're going to have to figure out for themselves what this means for them. And also who's going to support it. And I think all of those norms and things are discussions we have to have. But I think we all want to get everybody lifted up with these great tools, and there will be a mix of skepticism and positivity in there, I suspect.

It's always a mix of things, isn't it?

Yeah.

I'm—

so I just came from this CISO CAB forum we run in New York, and through some kind of weird scheduling quirk, I was assigned to facilitate a session on shift left. And right before I started screaming, I'm not an AppSec person, I don't even play one on TV. It was too late. And I'm in it. And I am with a bunch of leaders who deal with AppSec every day. And here's me who is like, can you maybe write an IDS rule instead? In any case, so this aside, the topic of process friction came up quite strong. Like Heather just mentioned science. And I'm like, Heather is right. Science is there. But is procedural readiness there at many companies? People say, Okay, so under what conditions is your automated scanner inside your CI/CD pipeline would stop the release? Okay, so you have conditions that are agreed with your big business leaders, and 1, 2, 3 happens, you stop the release. But then people escalate, scream at CIO, CIO screams at CEO. I don't know how they run these things specifically, but then it's unblocked. So, that's what makes me nervous. Science is there, but is organizational readiness? Care to answer this one? Because we have other fun questions. If you want to skip this one, it's fine, right?

No, let me say this. I think there's always a very natural tension in business between velocity and safety, I don't think anybody would ever say, I want to ship unsafe code. All of these conversations come down to what's the solution for doing that?

Mm-hmm.

And I think once the solutions are on the market and ready, and more importantly, once the developers are coming out of school having used them in school, they're gonna want them, and we can make them cost efficient, I think it'll be a win-win for business because I don't think the CIO wants to ship unsafe code. I don't think the marketing team wants to ship unsafe code. It's just we haven't had great tools for doing it right. And I think once there's enough proof points, case studies, wins, and things that get public and we can all share what's working well, I actually think over time you'll see this change. And more importantly, the problem is once you invent seatbelts, the regulators decide seatbelts are mandated.

Hmm.

If this really works, you may see in the future, I think 10, 15 years, probably quite quite a ways down the road, you might start to see insurance nudging the market. You might see regulators nudging the market. And you might see more of it just becoming the norm.

OK, so that's a great segue to the next thing we wanted to ask.

Yeah, to the regulators.

Yeah, the regulators.

I'm kind of slightly dying to ask this one a little bit, not because it's another triggering question, maybe not for us, but for some people. And the story that I remember goes to the early EDR days, when a certain EDR vendor that is only cloud-backed, like a well-known EDR vendor that cannot deploy on-prem, does not have an on-prem option. I was selling to certain people in a certain region, and they were saying, whoa, your EDR is backed to the US cloud. We can never use it. And the question that I asked the founder of that company was, okay, so, but if you are better by 10%, or if you're better by 10 times than the next best on-prem EDR, would they go with regulations, or would they go with risk reduction? And I used it as a kind of a triggering question at the time. And oddly enough, in those regions, everybody said, we'll go with regulation. We don't care if we use the 10% effective EDR while the 10 times more effective EDR is available because regulation. That shifted something in my head. It made me think that people are willing to, well, have their business die. I'm sounding sensationalistic a little bit, but follow that mandate.

I think as risk managers, we're trained to look not just at the risk in front of us, but the risk in the rest of the business too. If you are in a regulated environment, ecosystem, and sector, the risk of the regulator may be higher than your cyber risk.

Yep, that's what they pointed out, exactly.

The regulator is the launch, right? They'd get to decide whether you exist or not, right? You know, in those environments you have to lean into like, yes, we're gonna do that. I think they would also love if they could use some of these solutions. But this raises all kinds of questions about sovereignty and, and things that are really quite complicated. But I think, you know, we have to, as cyber people, not get so focused on our cyber risk, but also understand businesses have other kinds of risks and we need to balance those out for them.

But then with AI, what do they need to change? If a certain AI innovation— if most AI innovation outpaces legislation, is there any kind of simple and impactful policy changes that others should make? It's not just like allow it, but like what can it look like?

Yeah, I think there it's really important for business to be speaking with their local regulators and talking through some of these issues because I think as these solutions become available and relevant, And we can put pieces together, you may be able to move the regulator on some of these things. There may be times when that's not appropriate and also not possible, just given the jurisdiction you're in, the sector you're in, the geopolitics of the situation. It's highly fragmented. It's highly complicated. There's no good answer. I think these are tough societal issues, and the only way to deal with them is to talk them out.

Okay, so I hate to do this because I think the 3 of us could talk about this all day and have a lot of fun doing it, but I have to ask our closing questions. One, aside from the awesome article that you wrote with, uh, Gadi and Bruce, do you have recommended reading? And then 2, do you have a piece of advice for CISOs facing this problem? And bonus, do you have a piece of advice for early career folks looking at how this is going to change the security industry?

On the reading thing, I feel like it's moving so fast, so there's not like any great new books out that sort of talk about this. But what I would do if I were a cyber person just kind of getting spun up on this is subscribe to some of the newsletters that are coming out because it's changing every single day. Daniel Messler has a really great one. And listen to podcasts like this. It's just changing all the time. And You've got to be in curious mode. You've got to seek out the information. You've got to learn what the kind of latest and greatest is. Join a book club, start a book club. Now's the time to pretend you don't know anything about tech and to learn it all over again because it's changing so quickly. My advice for CISOs and teams wanting to try out AI is that start with small pilots, start with proof points. Go slow. Don't try to sort of boil the ocean. Pick a few places where you can see in the industry peers being successful. Talk to your peers. Make sure you've got good governance over how it's getting used inside. And I think the workforce, so especially early career workforce, is going to end up driving a lot of innovation in enterprises because kids coming out of their university degrees, out of high school, are going to expect to be using these tools. I imagine it was a little bit like this at NASA when everybody moved from doing paper computing to calculators.

Hmm.

And, you know, there was probably a generational shift in the people who really wanted to do it with pencil and paper and chalkboards. With, I've got a calculator and we're just, you know, we're going to do it that way. And then computers came along and everybody had to retrain over this time. So, if I'm early career, especially if I'm in the SOC, get used to AI as an assistant. Get used to the idea that all of that toil that you had to do before, you've got a tool now to help you with the toil. Your job is going to become a little bit of a supervisory, and I don't mean that like you're a manager of AI, but like your job is to scrutinize what it's doing and to make it do what you need it to do for you. And that requires an incredible amount of curiosity and critical thinking and really challenging what you're looking at. And we'll see how things develop, but it's going to be a very interesting couple of years as we see all this develop.

I really like that answer about how their jobs are going to shift. We keep talking about this with different guests, and I think that Critical thinking and curiosity and scrutiny, that, that keeps coming up again and again. I really like that.

Exciting times.

Exciting times as always in the security industry. Heather, thank you so much for joining us today.

Thank you for having me.

And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast. By Google.

Security.