EP150 Taming the AI Beast: Threat Modeling for Modern AI Systems with Gary McGraw

Security.

This is the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button on your podcasting app of choice. You can follow the show, argue with us and the rest of our listeners on LinkedIn. Anton, we are talking about AI, and this was such a fun episode. AI is always fun, but this one I thought was especially fun.

That was especially fun. And I would say that we had a chance to talk to somebody who was hugely instrumental in like, building the whole field of software security. And I think now he focused his attention, his amazing mental powers to the problem of securing AI. And I think some of the lessons from the early days of securing software allows him to deliver magic in the area of securing AI, thinking about it, building frameworks. So to me, this is eye-opening from an information transfer point of view, but also from new ideas. Actually, some of them are old ideas.

Right.

But old ideas we can use As new ideas.

I don't know.

That is something that has struck me time and again in AI securing conversations is we see these old problems in new clothing. We worry about prompt injection attacks, but prompt injection sounds a lot like another kind of injection that we know quite a bit about as an industry. And so it's curious to me to see the kind of repeating fractal nature of security problems. And so maybe with that, let's turn things over to today's guest. Today we are joined by Dr. Gary McGraw, founder of the Berryville Institute of Machine Learning. Gary, first of all, that's such a pleasant sounding place. Like, I want to go to Berryville and pick machine learning models off the vine all day. That sounds great. We're talking about the intersection of AI, ML, and security today. And one of the things that I think our industry likes to do is get really hyped up about stuff. And we get really hyped up about new things and securing those new things. And one of the memes that we see these days in the industry, one of the tropes we see in the industry today is that we're behind on securing ML and AI. And while I'm sure this is great for people trying to raise a round, it's maybe not the best thing for security leaders who are trying to allocate their own investment internally of what they're going to secure next. So how do we think about this problem? Are we really behind on securing AI? What's the current state of things?

No, I don't think we're behind, but I think—

Oh, thank God. So we can end the episode here.

Are we done?

Are we done?

We're done.

No, say more, Gary, please.

Don't quote me on that. No, actually, my belief is that we have an opportunity right now, and the opportunity is to think about security while we are building applications out of things like foundation models. And so there's some time to do some stuff right. Now, of course, When you're building security in, you really should start at design. And those are the things we've been talking about at Berryville Institute of Machine Learning, but I don't feel like we're so far behind the 8-ball that we're never going to catch up. And I will say this, I'm one of the guys who helped to establish software security as a discipline some 25 years ago. And machine learning security feels to me like software security felt 25 years ago.

What I love about this answer is that I feel in you a kindred spirit, because what brought me to Google 5 and a half years ago now was the opportunity to build security into cloud. We were early days of cloud 5 and a half years ago, and we had all these opportunities to bake security in, in ways that we never had before. So when you look at the stuff I've shipped here, like the built into our hypervisor introspection of memory, that's really built in, not bolted on. And so when we look at AI, what are some of the opportunities in AI to build it in rather than bolt it on? What's that look like here?

Well, that's the tricky bit.

Oh yeah, sure.

Yeah, AI is new from a security perspective because many of the risks associated with building these systems are tied up directly in the data that the systems are trained up on. So as technologists, all 3 of us here are really good at understanding tech stacks and thinking about building security features in and stuff like that, but we're really not so good at understanding enormous datasets and the risk that those drag into our models. Because machine learning is about making a machine become the data. The learning process means that the machine just eats all the data and becomes those data in some sense.

You are what you eat.

You are what you eat. And when the datasets are enormous and we haven't had time to pick through them looking for all of the garbage and junk and poison and bad ideas, Then those bad ideas are going to be built into our machine learning systems. So we have to be extremely careful about controlling data risk. And that's new for all of us.

Is there a food pyramid for AI models?

Somebody was talking about BOM for SBOM. Well, I guess DBOM would be the answer. I don't know. Like it sounds a little weird, but—

I like food pyramid better.

Not the bomb.

Oh man. So do I. I'm not a fan of either one of the metaphors. The SBOM strikes me as flavor of the day, Anton. It's just like, whatever the software security guys are talking about today.

Alan, you hear that, man? Flavor of the day.

It's true. Flavor of the day. And it'll help some people do some good things like, I don't know, dependency checking, which we've known about for, oh, I don't know, 40 years.

Longer than I've been alive. Yeah.

Exactly. And we should in fact be doing that. So I'm not opposed to SBOMs. I just don't like what happens when some sort of Meme or idea, you know, meme in the old Dawkins sense, takes all of the oxygen in the space. In some sense, you can think about that from an ML perspective when everybody was talking about adversarial examples. All the oxygen was taken up in the space, worried about those attacks. And as you know, that's sort of a minor detail. It's not the most important thing out there. There are plenty of other things we gotta worry about, including this notion of— understanding the data that we eat before we eat it.

What else do we need to worry about?

Well, the biggest problem now, in my view, that's right at the moment has to do with foundation models and large language models. And that is eating your own tail, or what Ross Anderson calls recursion pollution. I call it recursion pollution. So the notion is that we scrape the net to build our foundation models. We have enormous datasets like 14 trillion points. Then we produce all sorts of material which we put on the internet. And so for the next generation of stuff, we eat the stuff on the internet, and we really like the stuff that we produce and put on the internet as an AI model. So we end up building this kind of recursive pollution loop. If you read the Anderson paper, which is a superb paper—

When hasn't he written a superb paper?

You know, Ross writes good stuff usually.

Yeah, he does.

This is Shumilov doing most of the work, and Shumilov has written about sponge attacks and This recursion stuff, we pointed this out as a serious problem in our work 3 or 4 years ago in the architecture risk analysis of ML systems. But it's become more important because of the way large language models are trained and where the data sit. And this loop is a different sort of gigantic planetary-wide loop now. And we need to think about that.

Wasn't there something, examples already with Reddit? using answers from AI and then they're showing up on Google. And then there was already some kind of weird story recently about the circular pollution of that sort. Like, it literally happened already.

That's right. It's happened multiple times. There's also a Google example that may be apocryphal, so we won't talk about that one. But this notion of the loop is something that we've known for a long time is important. Another way to think about the loop in a much tighter micro sense is The feedback loop that you get when you have an online system that continues to learn and continues to be trained while it's fielded. The obvious example there is Tay.

Oh my God, that was a scary one. Tell the audience who maybe aren't familiar with Tay so immediately what happened there. What's the short version of that?

Right, so Microsoft built a bot to live on Twitter, now known as X, and the bot was on Twitter. I think it was turned on for maybe half a day, maybe 2 days. It was a very short period of time. And it was still learning. So it learned to become a xenophobic racist. I don't know if I'm allowed to say this, but asshole. Troll. And that's bad. Yeah.

Like a bunch of jerks turned Tay into a Nazi really quickly.

Really quickly. So that's the feedback loop at a micro level. The feedback loop at a bigger level is much more subtle and hard to discern, but much more important as well. It turns out that facts are facts, science is science. We do need to cite our references, and peer review is a good thing. Some of the modern world is cutting against that stuff, but it's not the first time ever.

Why can't you make the model do peer review?

That'd be great. I think we could make the model do peer review and everything would be rejected.

So before we go there deeper in this exciting field, I am not going to defend the DBOM, SBOM. But like, if you do say that it's about data, and I made the same point very often, that if you program with data, you should treat data like code. That to me is like the type of logic. But ultimately, if not just saying, hey guys, just do data governance better, which is probably not going to go well with people, what is the answer if we are talking about, what is it, 38 terabytes in some cases?

Yeah.

If your answer is data governance, you're going to put everybody to sleep and then they'll have very secure models because they were sleeping and didn't build them.

Now wait a minute.

You guys just said that word 3 or 4 times. And I didn't even get to say anything yet.

Okay, sorry.

I'm not saying the G word. I'm just not gonna say it.

What do you say, and what's your answer here?

So here's what I think we need. We need some sort of science of representation. We need to understand breadth and depth of data. We need to understand distributedness. We need to understand sparseness. And all of those aspects of representation of data should inform Our view of a dataset. And by distributed, I mean something like Penticanerva's sparse distributed memory sort of thing. We're thinking about this hard at Bimble, but, you know, it's just a tough nut to crack. So we're still just thinking a lot, and we haven't come up with a great solution. But it would be superb if there were some way to, at the largest scale, like zooming out from planet Earth, say, hey, look, there's 7 continents, right? So, we need to think about those 7 continents when we put them in our machine and become the Earth data. And so, that's the sort of thing that would be great to work on. I was just talking to a bunch of data scientists yesterday, and they agree that we need to work on that, and nobody's working on that. So, it would be superb to have some progress made along those lines.

It is useful to think about it, but I'm just— I'm not gonna say data governance 3 more times, but I would say it once. I think your answer applies to both cases of training on the data you control and training on the data you don't control.

Yes.

Which, of course, the previous thing I promised not to say anymore doesn't apply to the data you don't control.

Data poisoning is a real risk that we've known about for years.

What's data poisoning? Tell us about data poisoning real quick.

Data poisoning is the idea— I mean, there can be pre-poisoned data just by reference of being crappy data, but the data poisoning, let's assume that we have an intentional attacker who is purposefully screwing up the data so that when you eat it, it makes your results bad. And there are many, many papers on data poisoning. And it's not something new. It's something that we've had around in security forever. If you think about Dave Wagner's work with SSL back in the early days, what aspects do you control that go into the random number generator? Aha, process table, blah, blah, blah. That's the same sort of poisoning thing. When you're assuming that your data are clean and they're not, and not only that, but an attacker can get to the place where you store your data or eat your data from, then we have a problem. And that's what data poisoning is.

Got it. Okay.

Moreover, just coming from an event recently where somebody pointed out an example where you can pre-poison the data before your opponent is using the data or even building the models, which to me is like, I'm getting paranoid thoughts from that, frankly, because it was sort of a slightly geopolitical example, and I'm not going to share the whole example, but it was about somebody poisoning the data today with an idea that you would use it in 3 years for AI, which just freaked me out. This freaked me out.

I'm totally with you. I think that's not an unreasonable thing to think about and not too paranoid.

Not too paranoid? Then what's paranoid?

I do believe that we're going to have some trouble thinking about data out there in the world That seem to have occurred after 2021. And the question will be, did it get created by a machine learning model or a human being? And we just will not know. So after 2021, all bets are off. Now, of course, we can go pollute timestamps and corrupt old data too, but that's just an interesting phenomenon. So we want clean data piles, which means we need to control our data, which means we're beginning to build data moats. And if we build too many moats, we're gonna have data feudalism, and that's not so good either. So on the other end of the equation is everybody's hoarding their data and nobody's sharing, and we end up with machine learning models that are kind of paltry and underdata-resourced. And so you gotta balance all those things out. It's just like every other security engineering decision that you make. How much data of my own should I eat? Is that enough? Is that too paltry? How much public data should I eat? What about poison? How do I know about the aspects of my enormous dataset of the sort that we were talking about before?

This problem of data that wasn't ML-generated reminds me a lot of mining shipwrecks for low-background steel. Steel produced after 1945 has atmospheric contamination that includes radioactive particles.

Right.

And so if you're building very sensitive scientific equipment, you can't have, in many cases, is that steel in your equipment. And so people literally mine shipwrecks that have been on the ground since before that time for that.

And so it's beautiful. I love that. I'm going to steal it.

Yeah.

Wow.

Yeah.

Awesome.

Yeah. Low background data for your training. Low background steel for your scientific equipment.

Right.

I want to continue on this path about the data poisoning and maybe widen the aperture a little bit. How should people think about threat modelling for their AI systems or their systems built with AI tooling somewhere in the chain?

I think that's a really apropos question.

Thank you.

My favorite one so far.

Favorite question award.

Yay.

I've always been a fan of thinking about design flaws in addition to bugs in software security. So I've been talking about architecture analysis and threat modeling for 25 years. I think that we have made some progress thinking about what risks are in ML. You can look no further than the architecture risk analysis of generic machine learning systems that we produced at BIML 3 or 4 years ago, where we identified 78 risks. And that doesn't mean you have to control each one of them meticulously. What that means is you need to at least be aware of those risks while you're doing your design. And that's what threat modeling is about. So threat modeling says, hey, I'm building something like this. What's happened in the past? What do I know about the risks that I'm going to drag into my thing by Using this stuff right here, this particular design. And I'm happy to say that we put our stuff out for free under the Creative Commons. People are using it. I know it's being used internally some places at Google, and that's really cool. I think that if we can automate that into some automatic tools for threat modeling, like maybe Irius Risk has a threat modeling tool that they're producing. I know that they're building a pile of risks or threats to put into their model based on the BIML-78. And that's the sort of forward progress that I think would do everybody a great deal of good.

But there are differences between threat modeling for these types of quirky— I would use a technical term here— AI systems and the traditional complex enterprise systems. So it's almost like I had a bit of an obsession about the differences between securing AI and securing traditional systems. So maybe I love your view, given your software roots, so to say. Go through kind of like difference in threat modeling and maybe broader differences in thinking. Ultimately, if I have CRM or some other big, hairy, messy enterprise system, there are security challenges. If I have a system with AI, I have some of the same challenges but also some different ones, right?

That's right. And the difference we already covered, and we were all nodding along, and that is data-driven risks that come with the data. Now, I do want to point out one important thing, and this is a distinction that a lot of practitioners are forgetting about in their huge rush to adopt AI for any reason or no reason. And that is, we use machine learning when we do not know how to solve a problem specifically. That is, we can't write a program in a normal way to solve that problem. Instead, We know what we wanna do, but we don't know how. And we use machine learning to build that what by becoming the how. And we should never forget that. So if you know how to do something, for goodness sakes, do it a normal way. And if you don't, and if you just have to worry about the what, then machine learning is perfect for you. But it drags along some risk and some of the risk has to do with what are in the data. that you are using to build the machine.

So on that vein, I want to ask a slightly different question. And this is one of Anton's favorite questions and maybe one of his favorite concepts is to pick at what's the difference between securing AI and securing any other complex data-driven application inside an enterprise? Are they really that different?

No, it's not really that different. Yes, I don't believe.

I like that answer, actually.

Yeah, I really don't believe it's that different, although it's harder in some sense because As the guy who helped to invent static analysis and build those tools to look through your code finding bugs like a zillion trillion years ago, which everybody uses now, I can assure you that I would like nothing better than an automatic static analyzer for a machine learning system. Ain't gonna happen, guys.

Ain't gonna happen.

Ain't gonna happen.

Ain't gonna happen, so don't look for it. We can use that as a metaphor, but what is the static analysis of machine learning?

People show up and say, give me a scanner that I can like spray the model with that. and know the risks, not even the security challenges. And we're like, have you— let me start from the gas cloud and dinosaurs to explain why it is not going to happen. I love it. But the desire is there.

Well, look, there's an even worse thing, which is even stupider, and it's also from stupid security land. And that is we can somehow dynamically test our way out of this problem. So when the White House says, we've got everybody at DEF CON red teaming machine learning and we're just gonna fix it after they find it with prompt injection, is the dumbest thing I have ever heard in my life. That's just dumb.

Wait, wait, but because you won't find all answers or because you shouldn't test? Because I think you should test, right? You should test.

Oh, I think you should test too, but I don't think we can penetration test our way out of this, especially when we're trying to shoot through a tiny little slit at the target.

Yeah.

And so, the notion that we can pretend that that is the way to do security engineering is just as dangerous as the other notion. And it's along the same lines. I remember in software security when it was all about just breaking systems and pen testing, and nobody was talking about, hey, it's bugs back there that are causing many of those problems and/or flaws. And we got to fix the software, not just put a thing to stop the attacks, Like a firewall, but we actually have to fix the software itself. And I think that we're in somewhat similar of a situation with machine learning right now.

I'd agree with that. I think pen testing is so funny. I used to work for a guy who referred to it as a fundraising activity because the only thing it was good for was causing your board or your executives to give you more money to go solve the problems. Yeah.

Well, Phil probably said that too. So you still work for a guy who says that.

For better or worse, I don't work for Phil, just Anton.

Oh, okay.

That's legitimate though. Wait, I mean, that's legitimate. If people are not motivated enough, you can PT and then they're motivated. So like, that's not really a corrupted use for PT.

No, no, it's not at all.

Well, but look, if it takes up all of the oxygen once again in the room and we spend all of our time talking about red teaming and stupid prompt injection stuff, which is highly entertaining and fun, but not really much to do with security engineering, then we're gonna find ourselves at the end of the day, With the same problem that we started with.

Okay. Yeah. So we're coming up on time. Anton, do you want to pick one more question to ask Gary before I go to closing questions?

Yeah, I actually want to— well, I want to have Gary wax poetic about avoiding the mistakes, but—

Oh, I want Gary to be here all day. This is great.

Do it again, guys. I'm here all day.

No, no, no, no, no. I feel like I want to go to frameworks. I want to go to frameworks. Like if something is—

You want to go to frameworks? You're such an analyst.

Yeah, exactly. Thank you. I mean, I do treat it as a compliment, but Sorry, can I have 2? Let me have 2. So one is kind of the framework for thinking about it, but one particular, like, how would I structure my thinking if I'm coming in the field of securing AI? But there's a sub-question that I'm gonna like sneak from under it.

Okay.

Some people build, tune AI, and other people consume AI.

Yes.

So there are clearly differences, and I would want your framework you're gonna create in the next 3 seconds to include that.

I'll grant that's a very good question.

Well, here's the good news. So we built a generic model of machine learning that includes 9 components and also takes a system view, so that's component 10 in some sense. And we did a risk analysis based on those 9 components. That applies really well whether you're using or building a system. But if you take the generic model and you turn it to something specific, like let's turn that same model towards foundation models for LLMs and the use of those foundation models by people building applications, then what you end up with is a black box around 5 of those components that are controlled by the vendor that built the foundation model, whereas there are still 4 that are outside where there are plenty of risks that apply. And you have to think about things like how, as a user of this system, should I demand or request metrics and measurements and understanding of the black box so that I can choose between, say, I don't know, Bard and ChatGPT or GPT-4. How do I make that decision right now? Right now, the decision's being made by who's playing golf with who, and that's not the best way to make tech decisions, as you guys know.

Usually not. Okay. I like the answer. I hate to do this. Our traditional closing questions. Do you have a tip to help people improve their AI security outcomes? And 2, recommended reading? And I suspect for that second part, there's a big list.

There's a big list. Let's start with 2. So if you go to berrivillaml.com, which is our website, There is an annotated bibliography that talks about hundreds of papers where we have a comment about all of the papers that we've read in our research group. But more importantly, for the question at hand, we have a top 5 list that we curate.

No way.

So if you're interested in how to get started, read the top 5 science papers that Bimmel says these are the top 5.

Amazing.

And it'll give you a good broad understanding of what to do in the field. My one piece of advice is just going to be generic pablum, which is that if you think about security while you're building these things, you're going to be better off than if you just put off security for later or think security is some magic fairy dust that you can sprinkle on your system or think that you can pen test your way out of this with red teaming. We know better than that as a field. We're making good progress. People are doing good work. You guys are doing good work. BIML's doing good work. There are people at OpenAI actually working on this stuff. We just need to get the word out and use the tools that we're building, and we'll make progress.

Gary, I love the optimistic note of we'll make progress. This has been a fantastic, wide-ranging conversation. I can't thank you enough for joining us today.

My pleasure.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.