EP256 Rewiring Democracy & Hacking Trust: Bruce Schneier on the AI Offense-Defense Balance

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Detection, UEBA, Risk, all kinds of good stuff in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy this content and want it delivered to you piping hot every Monday, please do hit that subscribe button in your podcasting app of choice. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we have somebody joining us today that you could not help but gush about. Yes.

And it is not just the celebrity worship. I promise you, it is genuinely the admiration of some of his ideas. I would say that this to me is about the thoughts, not about the person.

Well, I think, listeners, it's pretty funny. When I first met Anton, literally the first time I met him professionally at Google, I looked at him and I said, oh, you're that Anton. And it was because Anton is a relatively famous guy, and at the time, just some PM at Google. But I had that, again, that feeling of meeting a celebrity when talking with today's guest, which was delightful. I first came across Bruce's name when I read Cryptonomicon in middle school. So that's how far back this knowledge of person goes for me.

But here's the thing. I had the meeting the celebrity vibe as well, even though I met today's guest, even though you named him already a few times at conferences. But this to me probably was the deepest conversation we ever had. And so I had this whole celebrity admiration as well. But again, to me, the ideas, not the status, is what drove the conversation. To me, we are probably not, I mean, the classic celebrity worshipers. We don't even follow many on Instagram, probably.

Listeners, my conception of Anton's Instagram is basically huskies. And I think that's about it.

Correct.

Yes.

Huskies. And, you know, maybe there are celebrity huskies. Okay, now we are really going off topic.

Okay, well, we are way off topic, listeners, but I want to turn things over to today's guest. And I want you to approach today's episode as both technical and societal conversation. And so without any further spoilers, let's turn things over to today's guest. Today we are joined by Bruce Schneier, Professor at a couple of different places this year at University of Toronto, and joining us to talk about a book that he's got out and the topic of AI and society. Bruce, it is a pleasure to have you on the show. Thank you for joining us.

But I want to stop in there and say I'm not really a professor. I mean, I like being called that. My students call me that. I teach, but I'm an adjunct.

Okay.

Basically means they pay me next to nothing, but I get to attend zero faculty meetings.

That feels like a good deal.

But it's called adjunct professor. I mean, it is called adjunct professor.

I don't think of myself as professor, but yes, I suppose you're technically right. Probably delete this entire exchange.

So it's also just— it turns out I'm going to go into gushing mode instead of asking questions. Often we have guests, and Tim would say something like he knew the guest since his childhood. And I say, oh yeah, I know he's an industry colleague. But you are, in fact, both of our childhood heroes. Both of our childhood heroes. And that was kind of one of those examples where it's like, oh, Timur says, oh, it's Bruce Schneier. And I'm like, yeah, right, it is Bruce Schneier. So we're like—

And this is your way of saying that I'm old.

No, no, no. More wise than old. And you knew most of the secrets of security roughly in the '90s. So this is kind of amazing.

All right, listeners, we got a lot of Anton gushing, but we need Anton to ask a question.

Fine. Okay. So the question is, it's a question that bothers us. quite a bit. It's about AI and cybersecurity because people show up and say that AI mostly benefits attackers. And the result is that some kind of apocalypse is coming. So in your opinion, do you believe that AI is going to end up being a net improvement for defenders or attackers? Or maybe you have some kind of it depends answer.

Yeah, so this is actually the hard question. Clearly, it's going to be an arms race. Clearly, it's going to go back and forth for several years.

Mm-hmm.

You know, take something as simple as AI vulnerability finding, right? So we got a lot of papers, a lot of research in AIs finding vulnerabilities in code. Now, sure, the attackers can use that to find vulnerabilities, build exploits, attack systems, maybe automatically. That's bad. Defenders can use that to also find and fix vulnerabilities. That's good. The imbalance is That when the defender fixes them, they're gone forever. So you can imagine a future where these AI vulnerability finders are embedded in the compilers. They're part of the development process. And we are all producing basically bug-free code, not really, but to a great extent. So for that particular thing, long-term benefit is the defenders. Now, short-term attackers have the benefit because there's all this existing legacy code. Which will be vulnerable. So you can play that out in a variety of scenarios. My guess is that AI will enhance power both attackers and defenders. Near term, it net benefits defenders because attackers are already attacking at computer speeds, and the ability to defend at computer speeds is really valuable. We're not even there yet. We're almost at step zero. where the attackers are using them because hackers are just more agile than defenders, right? They don't have any bureaucracy. They don't have any procurement. So we are seeing the rise of automated attacks and ultra-powerful script kiddies through AI tools. We haven't really seen good defensive AI tools yet. I mean, I went to the RSA conference earlier this year, and everyone's got an AI strategy, but there's not a whole lot of AI tactics yet. But I think that's going to change. But long-term, my bet is on defenders, but it's going to be a rocky road getting there.

I am super in love with that answer. And no, it's not a continuation of my gushing, actually. It is that it follows the same theme that Phil gave us a couple of years ago. And his statement about it benefiting defenders in the long term aligned with yours. He has different arguments, but We felt like over time, the belief in that statement has decreased. But I think I like how you reenergized it. And I like the point about defending it at computer speed as well.

And the real thing is we don't know. I mean, I look at AI in many applications, and there are regularly these arms races. I mean, so I mean, I'll just give a random example. In my book, I talk about AI being used in the courts. And Brazil is an interesting example where AI is being used to make the court system more efficient. And you can see the difference. But at the same time, AI is being used by attorneys to file more lawsuits. Oh, no. So what AI gives, AI takes away. And I think you're going to see that again and again in so many areas.

So I want to talk about your book and talk about— it's called Rewiring Democracy, co-authored with Nathan Sanders. Kind of the same question. Do you think AI is going to be more helpful for, say, there's the forces of good liberal democracy or the forces of corruption, illiberalism, and authoritarianism? Just say those are the two sides of the world.

That's a scary question. I shudder when you ask that, by the way. All right, right.

So the answer is yes.

Oh no.

AI is a power-enhancing technology. It enhances the power of the humans that use it. You know, if the humans using it want democracy to be better, then AI will assist in that. If the humans using it want democracy to be worse, authoritarianism to be better, AI will assist in that too. It's interesting to watch. I don't think there's a natural inclination. The interesting question to ask, more along the lines of attacker versus defender, is, does AI centralise or decentralise power? Oh, right.

Okay.

That's interesting. Now, we can imagine that being done in both democracies and autocracies. There, I think it depends a lot on the application and how it's deployed. Oh, it's an interesting example. Attorneys are starting to use AI in all sorts of things. And there are dozens of startups building AI assistive tools for attorneys, from everything like doing precedent research, writing briefs, analyzing the facial expressions of jurors to see if they're paying attention. There's an enormous amount of money here. Question to ask is, will those technologies make the best attorneys even better, or will it more broadly spread expertise and make the average attorney better? If it does the former, it concentrates power. I think it's bad for democracy. If it does the latter, more people have access to good justice, and that's good. And that's going to be very specific to attorneys using AI, doctors using AI, playwrights using AI. AI in this application, they'll all be different. And it's likely, again, to go back and forth before it settles on something. But I think a lot about— when I think about AI in social society, in democracy, about power. Does it concentrate or distribute power? The most important question to ask.

I really love that. And I think— sorry, I start every response. Tim, tell Bruce that I'm not normally like this, by the way.

He's not normally like this. He's normally kind of a pain in the butt.

What the hell? Yeah, that's fair. But it's the decentralization argument to me is a really curious version of it depends answer. But it's not just like a flat out it depends, but it's the centralization of power, Decentralization. But let me ask an odd side question to this. What about the— so, you don't believe in any kind of good, positive centralization of power then? Like benevolent, wise bastard on top?

So, a couple of things. So, certainly, I mean, that destroys capitalism, right? I mean, the single best thing we could do in the tech landscape is to break up the big monopolies, like by far. They're too big to fail, too big to exist, too dangerous, too powerful. They convert money into policy to make themselves more powerful. So, no, I don't want that. I don't want that in democracy either. So it's an interesting thought experiment to ask is, what would it be like to be ruled by machine? I mean, would that be better? And let's postulate an AI that is good enough to figure out the right answer. That it knows everybody. I'm making this up, right? It knows everybody, kind of runs the election in its head, and it says, you know, if we did it, this is who'll win. I'll save you all the trouble.

I think Soviet Union in the '60s had ideas like this. I read some really old books about the birth of Soviet IT, and it had exactly that vision in the mid-'60s too, that it's the future of the, like, 1980s or '90s.

So the problem with that, I think, is that democracy's purpose is not just to figure out the answer. That the process of democracy, us humans doing democracy, is part of its value. And an AI that produces the right answer isn't better, even if its answer is superior to us humans. Having agency, going through the debate and the arguments and the voting and the politics, is part of the democratic system and is not just an algorithm to get the right answer. But it depends, right? If an AI is better at reading chest X-rays, I'm in. I don't care, right? I don't need it to be a human process. If an AI is better at figuring out where to mine for gold or dig for oil, fantastic. Better predicting the weather, great.

What if it's Better at deciding where to zone for more housing?

So that's more complicated, right? So maybe we want an AI as an advisor. So let's take zone for housing. It's a political process. It involves emotions. It involves people who have different desires of what the outcome should be. And it is inherently a compromise. I think an AI can do a lot of that, but I think the human compromising Is independently valuable. What I would do is I would set up a system where the AI is an advisor.

Hmm.

And like any system, there are human advisors, then there'll be AI advisors, and they will advise the humans doing the process. And we can certainly imagine ways it could advise. We produce a potential rule, and the AI will fast forward 5 years and simulate the effects. And maybe you could do that with 20 different variants of this rule. So these sorts of things, I think, give humans better decision-making capabilities. But I want the system to keep the humans in front. This would be really different than— let's imagine AI being a CEO. That is not a democratic process.

No.

In effect, a company is a sociopath. Its single-minded goal is to create profit. And we could imagine a sort of more sociopathic CEO as an AI. Seems bad for society, but I could easily imagine a company doing that. We are going to defer to this expert AI because it seems to know what the right things to do is. We don't know why. It's opaque. It just tells us what to do. We do it. We make lots of money. That's perfectly plausible. I worry about a world where those exist, but my guess is you can't stop it, so it'll happen.

That's right. There's no real button to say, turn the train to this other track or something like it. Sort of like an extrapolation leads you to what you just described. That's right.

And in our system of the profits are what drives things, it will happen.

OK.

So it's more about the adaptation. The adaptation adjustment is the question, not, oh, we're on the wrong track.

Right. It's going to be the arms race. We'll imagine that corporations do that, which means other corporations will do that, which means regulators will have to do that. There'll be AI-assisted regulation. I mean, we're in the United States where we don't really have regulation. But you go to a country like Europe where they actually believe government should constrain Private actors, then you'll see more of it. So, you know who comes out in the end? You know we'll watch it, but it's going to be this arms race.

I might posit the goal of regulations not to constrain private actors, but to make sure that the externalities are you know captured effectively and benefits are socialized effectively.

But yeah, yeah, I'm a little yeah. It was too bit too dark.

Yeah, that's right. And and I'll define it as the goal of the government is to establish the playing field. on which the market operates.

Yes.

Right. So, for example, the government says you cannot have a company that buys and sells human kidneys. That is illegal. 150 years ago, the government said, that great business you have sending 5-year-olds up chimneys to clean them is now an illegal business.

Right.

We have decided that that is immoral and you can't do it. This is perfectly reasonable for society to do.

Totally.

In any reasonable world, we would declare surveillance capitalism as an immoral business model, and you couldn't do it anymore. But once you do that, then companies innovate within the constrained space where you give them a market.

So I want to bring us away from 5-year-old chimney sweeps and back to a differently depressing market, which is vulnerability and exploitation and how we think AI might impact the forces there.

Yeah, in this show, in this episode, we're going to just wobble between high level society, humanity to hacking and exploitation. So we're going to wobble again.

I mean, it's perfect.

So it's a question kind of related to the first one, is the attackers-defenders value? If we look at specific dimension of exploitation, I have a machine where I type the company name and it works for a week. I'm using the metaphor from another episode, Tim, and then it just hacks the company and gives you a shell. So if exploitation is always cheaper than patching and building defenses, Why do you think we have a chance even in the long term? So, can we auto-patch at the same speed? Can we get people to agree to that? But also, a side question to this. My fear is that it's not just procedural constraints, but it's a human's kind of, maybe I'm more willing to be compromised than to adopt a process to do it faster. I don't know.

So, that's not a security problem. That's an economic problem. That's the problem of it is cheaper to be vulnerable. than it is to be secure. You solve that through regulation, right? I mean, that the market has decided, right, through its calculations, that vulnerability is cheaper than security. You want to fix that, you fix that through the market. The interesting question you asked about the automation of attack, and we're starting to see that, right? Anthropic found someone who was automating ransomware. There's a bunch of examples of automated attack, and we're going to see more of them. But then you kind of quickly zoom through faster than defense. Like, I suspect we're going to move to a world where there is equally automated defense and patching, where you're going to have these kind of living systems that are constantly monitoring themselves, constantly hacking themselves, constantly updating and patching themselves.

Right?

You know, because that same tool that the attacker can use, the defender can use.

Hmm.

And that's key. And the question is, can the defenders use them and deploy them in a way that removes the danger, removes the exploit? Now, I think there's a lot of work between here and there. You know, what company would allow— right? You know, you may imagine a magic large enterprise software company. sells you this big piece of software, and then you're going to now modify it every other day based on what your AIs say.

Mm-hmm.

Company's not going to like that. So we got to really rethink how software is promulgated, what ownership looks like, who can do the modification. But I think we're going to move to a world of equally fluid defense as the attack you just described.

Yeah.

So, this is actually pretty profound because I still see people who complain that a certain defensive technique is too high friction, too risky. And you tell them, if you corner them, you grab them and you say, do you honestly think that getting hacked is cheaper? If you really shake them, they say, honestly, yes, getting hacked is cheaper, which is really amazing.

That's right. Because it's someone else's problem. Because It's long-term because I care about my next quarter revenue numbers, all the things that the market perturbs.

Exactly.

And what we said earlier, that the goal of regulation is to internalize externalities. If we internalize the true cost of these security breaches, then it's no longer cheaper. It's cheaper because those costs are broadly shared around society. Think of, I don't know, The CrowdStrike breach that stranded me in Detroit because I couldn't get a flight home, right? I mean, I paid for that.

That's interesting.

Yes. Yeah. No, that's—

I think Anton and I are both beyond gushing and now in deep thinking and trying to figure out what to say next.

No, no, no. But I do remember this. I almost remember it may be a quote from one of the previous books because I remember that thought about costs being spread. So, I suspect I read it in one of your previous books because it struck me as very logical, but it's also one of those things that everybody knows, deep down, but nobody does anything about it. Like, if I'm compromised, a lot of people pay, just not me, or I pay a little bit.

That's right.

The full cost is borne by clients, society, random other people, partners.

And capitalism right now, late-stage capitalism, is full of these things, which is why it's failing so badly. It's full of these externalized costs, externalized risks, Any system where the upside is centralized and the downside is decentralized, there's too much risk in the system because the smart actor takes the risks because they're not paying the cost.

Right. And why wouldn't they?

Right. Why wouldn't they? Right. I mean, the CEO didn't do that, they'd be fired and replaced with a less ethical CEO.

Right.

Duh.

Right. Unless we regulated ethics into the requirements.

That's right.

Which is the whole point of democracy, is to do that.

Which is hard, right? I mean, democracy, I think of it as how we make decisions as citizens rather than as consumers. It's a path of collective decision-making, but it is one where we are in our role as citizens and not in our role as consumers.

I like that a lot. And this is a really weird episode for me because it's feeling a lot more like my days at Brown studying political economy than my days at Google doing security work. I expected this to be a security episode, not a political economy episode.

It takes all kinds.

It does.

Actually, it doesn't take all kinds. We just have all kinds.

Well, I mean, that's careful with that, because of course, if security is staffed by people who are trained in the university at cybersecurity only, I'm not sure we'll be better off, to be fair.

I think you're right. And the best cybersecurity people are generalists, do think about things very broadly. I mean, you know that. And so the people probably you work with, the ones who are best are the think outside the box people.

Oh, for sure. For sure.

One of the reasons AI is good at cybersecurity, it naturally thinks outside the box because it doesn't know what the box is. Box is.

Yes, exactly.

Yeah. If you're unaware of the box, it's a lot easier to think outside of it.

Which is why kids are natural hackers. Yes.

Yes. I've noticed that. And not only the school grades, but yeah.

Because they just don't have that box conception. Right. They're just trying to finagle an extra dessert out of you, and they'll come up with all sorts of things that you never thought of.

It's a very beginner's mind kind of argument. So I want to ask actually a question inspired by Anton's kid, weirdly enough.

All right.

So at school, Anton's kid will use the phrase AI to describe anything that's fake or made up or a lie. How do you see the impact of AI on society's ability to trust things? Because trust is so central to the justice process or the regulatory process. So how does AI impact trust?

It's interesting. I think a lot about AI and trust, and I've given talks about it. It goes a bunch of different ways. I worry less about deepfakes and more about the fact that it'll be deliberately manipulative.

Say more.

I worry about the fact that it is being designed and fielded by for-profit corporations for their own purposes. And its conversational nature will make it more manipulative. I mean, take Google Search. I mean, there are the results I want and there are results I don't want that people have paid Google to show me. The difference has gotten smaller over the years, but I can still tell the difference, right? There is still a notice like, you didn't ask for this. We're giving it to you because somebody paid. And I can see that. The conversational nature of AI makes that harder, right? So, if I ask an AI to book a vacation for me, is it doing what's best for me or is it getting some kickback off the back end by some company? So, it'll be, I worry, more manipulative, effectively manipulative, and it'll be harder to disclose that manipulation. So, I do worry about that. And I think that impacts trust a lot. But again, that's not the tech. That's the for-profit corporations that are controlling the tech. I mean, we can imagine an AI that doesn't do that. And it's just an AI that works for me, right? Isn't secretly a double agent working for somebody else.

Yeah, yeah. Like, it's basically— I can give you a perfect example. I have one of the Amazon devices, and it just got upgraded to some new version with better AI. And I asked some question. And the AI said, I can help you shop for that. And I said, thank you for being honest, but my question was more informational. And the AI said, well, I'm a shopping assistant. I turn everything into shopping. I mean, I'm paraphrasing, and I'm obviously not pointing at any specific company, but— Well, I can assume. But that's an example.

That's a great example of an AI that is working for the company first and for you second. Now, you asked specifically about— About generation of fake images and maybe fake videos. Now, that's nothing new, right? Manipulating photographs is as old as photography. Photoshop's been around since, what, the 1980s? What's different now is it's getting easier to do.

Mm-hmm.

And I don't know what effects that's going to have. My guess is the kids are all right, and that when you have created deepfakes of your friends since you were 12, you have a healthy skepticism. of things you see on the internet that don't come with pedigrees. So it's not, do I trust the image? It's, do I trust the source? Do I trust the New York Times? Do I trust a random friend on Twitter? I mean, that's what it really is. And my guess is we'll get there.

Hmm.

Already, I see in my students a very healthy skepticism of anything they see on the internet.

Yep. I worry more about older people who just were sort of, oh, it's in the newspapers. It's like, what? Right, right. They grew up.

Yeah, that it's in the newspaper, therefore it's true.

Well, yeah, I grew up in the Soviet Union, so I knew that to be wrong.

But then you have like the Sacramento Bee, which is a— it's a propaganda outlet with a newspaper-sounding name. That's on the web, or even The Onion, right? The Onion's a newspaper.

Yeah.

And people get fooled by The Onion, which is like—

They do.

Incredible.

But that's the point of a humorous parody source, right? That you pretend to be—

I don't know if AI— how much AI makes this problem worse, or if it makes it so much worse that it goes around the horn and becomes better because everything is—

That's interesting. Anton, I think you have one more tech question, and then I will close.

Yeah, I have one more technical question, yes. And this is— I mean, it's going to be a really odd one, and maybe It's about defense in depth.

All right.

When I think about this whole AI-armed attacker fears, I kind of sense that defense in depth is a really good argument for the defense, that if you have more layers, that we would— it would delay the arrival of the bad guy with AI.

Right.

Do you think— yeah.

Yes. The defense in depth, the cyber kill chain, I mean, all of those ways of thinking about the attack process as a series of steps. Tells us where to have interventions. And I actually just pulled that cyber kill chain up like a few days ago. And I'm trying to think, where are— I think there are 7 steps. How can AI improve each of these 7 steps? And I think there's an answer for each of those steps where AI will make the defense better. Is that going to mean that the attacker's never going to win? Of course not. But I think that's how we need to start thinking.

I would buy that. I would buy that it strengthens the defender's arm in all dimensions. And it's not just the layers and defense-in-depth argument, but it strengthens discovery. It strengthens the other parts of detecting early, detecting late but better. I think that makes sense.

Yes, I agree.

Tim, it's probably better to go to closing because—

It kills me to go to closing, Bruce.

Yeah. I mean, I can't even— It's just getting interesting. Exactly.

I know we could talk about the role of AI and power centralization and democracy, gosh, all day. I mean, these are literally millennia-old topics, power and democracy. But I have to ask our closing questions.

All right.

One, do you have recommended reading? We hate to leave our audience empty-handed.

And we'll, of course, feature your book anyway, right?

Yeah, of course we'll feature the book. And then 2, do you have advice to help people Get ready for how AI is going to really impact society.

All right, so recommended reading in line with what we're talking about. I'm going to recommend Malka Older's Infomacracy trilogy. This is fiction, and she postulates a world— it's a fascinating science fiction world where the planet's divided up into basically different cells, and governments which are larger than these cells compete to govern those cells. So you can imagine a government governing 30 cells around the planet, and they're not contiguous. And in this world, there is this kind of ministry of information that's in charge of making sure that information is accurate.

Hmm.

And so it's an interesting political situation, which interests me, how to think about democracy. It's a lot about trust and truth and information, which is also things worth thinking about. And they're really good reads. So 3 books. So that's my reading recommendation.

Perfect.

I like that. And then do you have advice to help our listeners adapt to the, the coming world as it will be influenced by AI?

The only advice I have is to engage, that this technology is coming. It's going to change a lot of things, and we really have to start thinking about how. And you're not going to understand how it's going to affect whatever you're doing If you're not engaging with it, if you're not using it, if you're not trying it out. And do that regularly because it changes friggin all the time. Stuff that was not possible 3 months ago is suddenly possible today. Stuff that's not possible today will be possible in 3 months. So keep paying attention. You're allowed to say this is bad, but you're not allowed to say, and I'm not going to even study it. Right. Because Saying it's bad from ignorance, especially for us techies, it's not good enough. I mean, have an informed opinion.

I love that answer for so many reasons. But Bruce, I want to thank you for taking the time out of your day to join us on the show. This was honestly— when Anton and I started this podcast, we made a list of people that we thought would make amazing guests, and you were, you were on our list 5 years ago of people we would have liked to have on the show. And I'm just so delighted to have you. Big thank you to my friend Shuman for connecting us. Thank you, Bruce, for joining us today.

Thanks for having me. It was fun.

And now we are at time. Thank you very much for listening and, of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.