EP215 Threat Modeling at Google: From Basics to AI-powered Magic

Secure running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior PM for Risk Analytics in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. Follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, we're talking about threat modeling, sharks with laser beams, teams getting along and having good feelings, Clippy. This is a wide-ranging episode.

Yes, but it's also very to the point. It's one of those super dense episodes where you would get a lot of wisdom Probably like a really high wisdom per minute percentage.

Yeah, listeners, don't be put off by Anton saying it's dense. I think that's actually like what he means is it's nutritionally dense, not hard to chew.

Correct. That's an excellent point. And I almost like made realize that dense sounds like it's inscrutable. No, it's actually dense in terms of value.

Yes. And maybe with that, let's turn things over to today's guest. I'm delighted to introduce today's guest. Joining us from Alabama, we've got Meder Eng, a security engineer in Google's Product Security Engineering. Metter, thank you so much for joining us today. I want to start off with what I hope is kind of a softball question, but could you walk us through Google's threat modeling process? What are the key steps, and what's Google-y about it? Maybe we could talk about that as well.

Okay, sounds good. And thanks for the introduction. I'm super excited to be here. For our threat modeling, I think, you know, well, first, I think in our industry, threat's a super overloaded word. But so threat modeling in the sense here is more of Really going back to the core question frame that you see a lot with the Threat Modeling Manifesto and Showstack's work with, like, sort of, what are you building? What could go wrong? What are we going to do about it? Did we do a good enough job? And so, within that framing, our process is typically, once we decide a product or area that we want to threat model, we sort of first set the scope of the work so it doesn't go way out of control. It's not too big, it's not too small to be interesting. And then identify sort of, like, subject matter experts that we work with and sort of talk to them to get an accurate description of the architecture, how it works, what the important components are, the data flows in between them, what kind of data it's handling, where the data sinks are. And then, once we have that, you know, solid architectural description, then we go in and start, like, enumerating threats about, like, what could go wrong with those components or the flows on the most important flows all the way through the system.

That makes sense.

So, some of our stuff is kind of sizable, right? And I think that I think I love the description. Description, of course, makes sense. But if I imagine some excessively complex system that we use for something and we need to threat model it, the task feels kind of very complex. Like, we apply it to many areas. Maybe we apply it to products, infrastructure, maybe even third-party relationships. How do we chunk it in pieces? Like, where do we use it and how do we apply it, especially for complex systems?

Yeah. How do you eat the whale?

Yeah. Okay. Yeah. I think it goes back to the scoping thing I started with. Like, in The way that I tend to think of it is, you have this sort of, like, graph of components that have all these dependencies, and, you know, you group them into certain, maybe a product or something, but they may have relationships. And so, within that, you may want to threat model sort of either bottom-up or top-down. Like, you may want to look at, like, a very high-level description of something and get a rough, like, high-level threat model, and then you can recursively drop down into, like, finer pieces. and go do more fine-tuned threat models throughout the details of the system.

That makes sense.

But that initial scoping part's super important, 'cause if you make it too big, the threat modeling exercise will go on forever. You won't get results out of it. So it has to be tractable. But if it's too small, like one component, then that's not very interesting, 'cause it's not really doing anything. It's not communicating with other pieces. So you have to sort of find that right spot where the scoping gives you the benefits that you're looking for.

I really like the simplicity of how you explain the 4 questions. They remind me of a CEO I used to work for who wouldn't ask, what if this goes wrong? He would always ask us, what if this goes right? And what he meant by that was, okay, fine, say you launch that, you're expecting this many. What if you get 100 times that many on day 2? Then what? And that was always a fun thought exercise, which I suppose is a form of going wrong if you don't do it ahead of time.

Yeah, if you're not, I guess, scaled up to anticipate that, I guess is the point.

I still have lingering doubts in my mind. I think that the description did make sense, but if something is— and again, I'm not an AppSec person. I keep reminding everybody that I'm not really— I'm at best an amateur in AppSec, probably at best, best being the keyword. But if the system is complex, you can say, oh yeah, we'll cut it in small chunks and then do threat model, but not too small because then it'll be pointless. I get that part. But if it's something very, very large, like, you know, system of data centers doing cloud serving or something, This whole splitting in chunks just still sounds hard.

I think at a data center level, it would be, like, because you would end up, you know, it may be too high-level to find useful threats, like, and so, I think you would have to recurse a lot into the subpieces, because when you're talking about a data center, I mean, one of the things I find interesting about threat modeling is, you know, I sort of changed later on to doing security engineering from a software engineering background, and a lot of things that I found in this world, it was more like how to hack your car, how to hack a web app, it was very domain-specific, whereas, Threat modeling to me is like a sort of a generalized technique that you can apply to a lot of places. And so with the data center thing in mind, like you can start with the software and, you know, recurse it all the way down to the hardware, to like looking at, you know, hardware components on the motherboard, how they're connected to pins, like hardware security attacks. You can look at, you know, even you'll apply the same techniques with threat modeling to the buildings that hold the data centers. So it is a very large problem and, you know, If you're looking at something like that.

I've heard some really funny and questionably true stories about data center physical security threat modeling. I want to pick at something you said, which is kind of the lifecycle of this stuff. You know, obviously, you could always get into greater detail with your threat model or expand your threat models to include more scope, but that's 2 dimensions. What about the 3rd dimension of time? How do we know when to go update a threat model?

Yeah. So when we do it is ideally keyed to the software development lifecycle. Most people, you know, I think the intent of threat modeling is to do it at the design phase to find the problems before you build it because they're more costly to fix after the fact. Everybody knows that sort of story. But we tend to also use it for things that are already built because we have things that have been around a while that need, like, a deeper look. And it's still, for things that are already in existence, a great tool for understanding the architecture and what the current weaknesses and pain points in something that is already there. So in terms of how they're updated, the approach that we really strive to hit is having a baseline threat model for things and then incrementally updating them through hooks that we have in the security review and design review processes. That, in conjunction with, like, working closely with our partner engineering teams, in some cases they'll be building threat models themselves while they're designing the things. And we may come in as from sort of a review standpoint.

Mm-hmm.

Is another method because like, there's, you know, as you pointed out, time and, you know, keeping updated is definitely a challenge. And then also like scaling it out's a challenge, you know, cuz even within my team, there's only so many people that can do so many threat models a year. So we really need help from, you know, where, where we can get it with NPSC, with, you know, engineering teams building the thing and So, this is a weird one because you mentioned that the original thinking was it's done in the beginning of the design phase.

But we probably met people who do it really well at the design stage and then write this whole big fat 800-page document, shove it into some desk, and then kind of not touch it. So, how do we actually operationalize all that? Because to me, maybe it's like this whole question number 4 we are asking now is kind of the point of the podcast, right? How do we actually reduce risk? How do we affect security posture with this?

Yeah, the method that you just described to me is just like waterfall software design all over again. You know, you write the big design upfront, throw it over the wall, and it goes out of date once you, like, test it or whatever. So ideally, I think if you had a proper baseline for the threat model, and then as it's changing, you have your incremental updates that, you know, produce the threats. And then it's sort of like that part in the 4-question framing, what do you do with this information? And so there's a lot of different things that we found to do with it. One, which is the more classic thing that folks focus on, is building mitigations to whatever the threat is. So you introduce some controls, like you adjust the software to try to, you know, limit the likelihood that the threat could happen or the impact. In some sense, threats never go away. You just have a, you know, a set of mitigations that help reduce likelihood impact, but there's always the chance that someone could find some zero-day or some unknown technique to, like, realize it. It's also interesting, I found, and that's more of, like, the defensive bit, but I think it's also interesting to use as sort of a map for, like, attackers' offensive kind of stuff, too. So they can look at it as, like, a roadmap because sometimes with threat modeling, it can be a little bit theoretical, like, oh, this could happen if this, this, and this.

Yeah.

And so for the things that we consider, like, higher impact, it's interesting to have someone go off and go, all right, let's see what we can do here. Like, if we can actually realize this, into a vulnerability, and then take that vulnerability and realize it into an actual exploitation path.

Hmm.

I have seen some red team reports at Google that really do keep me up at night.

Yes.

Yeah. Actually, I've had other red team reports that were worse than kept me up at night. I've had one where I was like, that's it. I'm done working for today. I'm going to go have some whiskey. That was scary.

But it's OT security. Yeah, we haven't— we probably should not touch OT security. No, I wouldn't touch that. Otherwise, we are all drinking whiskey during the workday.

So, I'm curious though, like you said, it can be theoretical, and I've also seen that happen a lot. What's your advice to our listeners to help take threat modeling from the realm of imaginary tabletop to really like, how do I communicate this to my teams? How do I get my software engineers to make better decisions than they would've otherwise?

So, with the theoretical part, I mean, it's always sort of tricky. Like, I mean, there's a spectrum there. Like, you don't want to get so concrete that you're actually pointing out a vulnerability that already exists, and then you don't want to be so general that it It's like, you know, the sun is flipping bits in some memory somewhere that causes the software to turn over or whatever.

Yeah, right.

So, I mean, to some degree, it's like getting familiar with the actual architecture of software, of what you're building and the properties there. Like, I mean, I found with me, it helps to actually use and understand the software that I'm working with to get more of a real sense. And then there's other data that you can take in from threat intelligence where people are building similar software and it's been exploited this way. And so there's a lot of, like, sort of real-world triggers on things that have actually happened blended with some sort of experience and intuition on working through these systems. I think that's helpful.

That makes sense. And I think that we've actually been specifically curious about how it's communicated to software engineers. And I think you answered that, but I think I recall the failure mode when security team sort of lives or makes the threat model and just, like, it stays with the security team and it never gets to developers. Like, it's kind of depressing because We're discussing the finer points about communicating to developers, while at the same time, there are whole parts of the world where they're never communicating to developers at all. And that problem never comes up because they're sitting on the CISO's shelf, in the CISO's inbox somehow.

Okay.

So, on this depressing take, I guess.

Well, with the ones that we typically do in PSC, we do them in partnership with the product team. So, we have a representative from, like, you know, someone that actually has a very detailed understanding of what cloud product we're building, and we do it as a collaborative effort because really they're best positioned to tell us how it works, like what is the architecture. And then together we can sort of do the threat enumeration and, you know, figure out things, what could go wrong. And we've, you know, worked with teams where this, you know, went spectacularly well, where they sort of turned the output of the threat model into sort of like a security roadmap to drive like—

Oh, that's cool.

The next iterations of the things that they were working on.

And I think that's also another interesting bit from that is that, You just say, we were collaborating with developers, and we all nod, and Tim nods, and Anton nods, and everybody nods. And it's like, that line alone is probably like a multi-year effort to build ties at some other company. Here you say, well, isn't it better to work together? We should work together. And guess what happens? We work together. Well, at some other company, I've seen people where to get to that line is a multi-year project hiring consultants who explain how to change culture and It's kind of amazing. This is like my moment of being proud that we're at Google, where it feels like this stuff is just, it just works. And it's amazing.

Yeah, it is amazing. I've experienced that friction in like other contexts. And it is amazing when you cross that line. And, you know, because I know a lot of this, like with threat modeling and security vulnerabilities, whatever, there's a lot of focus on the technical part of it. But there's a lot of great, I think, relationship building that goes in and building trust with one another that you're working together towards a similar goal to build like a secure product and things like that. that come out of these kind of collaborative projects.

Yep.

Okay. That part is kind of magic, but it's not magic because we have this new tool that does this, but because sort of the practices and people are just better in that sense.

I would be curious though, do you have advice to help teams get to that state? Because obviously there's such a big human component to this and some of that is cultural, but some of that is just interpersonal. So do you have advice to help people get to that nirvana?

In my experience, it's always easier if you have someone in the engineering organization that's already passionate. I think if you try to come lecture people and teach them about the benefits of security threat modeling, it's sort of, they're like, uh, like, or, you know, I'll hear pushback like, I'm not a hacker, I can't think like, you know, sort of a destructive, like, how are things break mindset, like, I'm a builder, that kind of thing. Like, and all of these are legitimate concerns too. Or like another one I hear a lot, it's like, I'm supposed to build the thing, test the thing, and now secure the thing, right? Like, I don't have enough time in the day to do that. So, I mean, These are all legitimate things, but I think I've found more often than not, if you have a pocket of people that are already passionate about security and sort of work through them to spread it organically versus a more like lecture kind of push-it-top-down approach, like that can be helpful.

But apparently, I'm sure there are cases where it didn't work even here. So, yeah, well, I wanted to ask this big fancy question about what are the challenges we face scaling this, but like, Are there cases where it just went wrong? I mean, I know we probably won't disclose too many details. It's a public podcast. But are there stories where we faced some challenge and things maybe didn't go exactly right, either scaling or building or utilizing threat modeling?

In the sense that we put a bunch of work into building a threat model, we thought we found all the things, and then next day we found some zero-day that burned everything to the ground?

That's an excellent example here.

I mean, sure. I think that's always a risk in this line of work. You know, that's why we have all these, I think, different work streams, what we throw at it across threat modeling, vulnerability research, security reviews, automatic tooling, whatever. Like, we sort of throw whatever we can get. So, I mean, I think that's always a risk.

Anything with scaling? Because some of the stuff, again, the scale that we have and where we apply this is quite staggering. So anything particularly challenging applying it to, you know, millions of things?

Yeah, I mean, that is definitely a challenge, and it really highlights the problem that you pointed out, Anton, at the beginning of the thing, like with the data center example, like how do you know what the chunk or scope or whatever? And we have like various things that we use to sort of create tiers and like sort of try to get coverage amongst like different tier, tier 1, tier 2, et cetera, but also really to try to be mindful of like the dependency graph between these things. Mm-hmm. Because like, you know, in cloud, and I think you talked about this in other podcasts, I think the one with Seth and threat modeling, like, it's a pretty hybrid environment where we have some things where we build on cloud, some of it builds, like, on native, like, prod infrastructure. And so there's a lot of dependencies that you have to be, you know, concerned about when you're looking to do that analysis. So if you have something that's, say, like, foundational, like GCE, like everything builds off of it, that's a good place to sort of focus your efforts on having, like, a solid threat model and making sure it's as Hardened as it can be.

So I hate to be the AI guy.

The team with the AI guy. Okay, let's make a moment of pause about—

Well, listeners understand the psychological struggle I'm experiencing at this moment. I have to ask it though, because I actually think the answer will be interesting against my better instincts. Are we doing anything with LLMs like Gemini to help people do threat modeling or threat model better?

Yes, we've been experimenting with that. Like, I've built some sort of prototype kind of things where you can give it an architectural description of, like, you know, what you're building, and it will help enumerate the threats. And I've had some, you know, generally interesting results with that, like, where it's more like on single or one-shot prompt kind of things. And there's cases where we've also been able to even use Gemini, like, Vision, where it can— you give it a picture of the architecture and it can decode things like the components and data flows and enumerate threats on top of that. So yeah, that is something that we're actively sort of pursuing using. I'm also very interested in sort of the types of approach that, for instance, we've seen with Project Zero with their Naptime work, where it was doing sort of vulnerability research using more agents. I think that would be an interesting approach for threat modeling as well, like a more agent-based approach where it could like look at documentation and use tools and all these kind of things to sort of, you know, give it more powers.

Yeah, yeah, yeah. That's, that's pretty interesting.

That part is very fun.

Yes.

It's almost like, To me, it does veer into the magical territory from auxiliary, supportive productivity improvement. At least in my mind, I think of 2 buckets of use cases for this, more like life becomes better and then life changes in dramatic ways. So this feels like it's closer to life changes in dramatic ways.

Yeah. And maybe you could have the similar in the ways we do with programming and Copilot and these things that ride along with you. Maybe you have something like that that rides along with you while you're writing your design document and Sort of like in real time points out threats, like as you draw architecture diagrams or sketch out data flows or whatever.

Oh, yeah. So you're actually designing and then some annoying bot shows up and sticks its digital finger and says, this diagram, it's gonna blow up. And you're like, oh, no.

Yeah, no, I can picture it now. It looks like you're accepting untrusted user input.

Over here, and it sticks its little digital finger in this spot.

And you're like, oh, yeah, that's a good point, actually.

Oh, God, I hate the idea of Cliffy doing threat modeling.

So to transition from this amazing magical land of Gemini helping you threat model, what about something more mundane? Like, I definitely see organizations stuck with threat modeling, and it may be the 800-page doc syndrome or never talking to developers syndrome or talking to developers, they never listen syndrome. I don't know. Again, not an AppSec guy here. Any advice from our— I should not say ivory tower, but from our well-run operation and lessons from that to people who are starting out? Like, what maybe you— what you wished you knew back in the day, or what you wished others knew to shortcut the journey? I'm kind of rambling a little bit, but I think that it's, uh, you get the gist of the question.

Yeah, for sure. I think for me personally, it was actually one of the hardest things to like really wrap my head around coming into it. But once I did, like, you know, it was a pretty good fit because there's a lot of things like if you're learning how to do like hunt for specific types of vulnerabilities and things like that. There's a lot of resources where you can do that in a hands-on way, like through these, like, whatever systems online and CTFs and various sort of, like, hands-on resources you have at your disposal. There's not a lot of that for threat modeling, so it makes it hard to practice. So, you know, most of what I learned was, like, sort of practicing and pairing up with people that were already proficient in it. And I know that's not an option, like, say, if you're just starting out in a company where nobody has done it.

Mm-hmm.

I mean, It really sort of like, if you can have a resource of someone like outside of your company to help you sort of get the initial thing off the ground and then, you know, sort of pair threat model like after that, I think is helpful. And then I think when I was first starting out, I would also just obviously read the standard literature of like the Shostak books and all that kind of thing. But then I'd go maybe find some open source piece of software, spin it up, threat model it just to practice on my own.

That makes a ton of sense. And you've kind of You've shortcutted me on the next question I was going to ask, which I hate to do, but I got to ask our closing questions. One, do you have a recommended piece of reading, maybe other than the show stack book? Because I feel like that's the first thing that comes to mind for people with this. And then 2, do you have a tip to help people go deeper on threat modeling maybe?

I can think of 2 books right off the top of my head that are relevant for me, at least with security. I really like Ross Anderson's Security Engineering book.

Yeah.

I love How to Solve It by George Polya. It's more focused on problem-solving techniques, mainly for mathematics, but I think it's got a lot of good knowledge in there about how to just think about and solve problems in a general way, which I wish we had more of in information security, or more like general methods. And I think threat modeling is sort of as close as we have to that now. To me, that's sort of a similar fit in terms of Tips? I mean, I think one part of it is really just getting sort of like hands-on, sort of back to like practicing, like spinning up open source, like maybe threat modeling yourself. If you're threat modeling something that's already in existence, like actually use the product, you know, get to know it, or maybe try to build smaller versions of it to see where the problems are. I think that's where I've found like a lot of insights and things in terms of, for me, sort of the Foundational skill for security engineering is understanding how things work. Because once you understand all the details of how it works, you can figure out where it breaks. And sometimes building, like, mock smaller versions of that and threat modeling it and things like that can be very helpful.

Sorry for a mildly naive question to this, but I have this picture of somebody trying to threat model an application they found somewhere or an open source application, and they've done it, but, like, how can they check? Like, there's no correct answer. They can't come to you and say, give us the actual character at model in answer. So yeah, again, I'm kind of expressing that maybe a very naive question, but how do I know I did it right once I tried it?

Yeah, it's hard. Okay, you're right. There is no, like—

Oh, you suck. You could quit too.

No, no, no, because there is no answer key. Like, you're right. And there's no LLM yet that's going to do it and tell you what— give you a grade. So I mean, I think you would— one, you could You know, again, falling back to sort of someone you know that maybe had more experience with threat model that could be like your proofreader. 2, I think there's probably applications you can find where someone's already pointed out the weaknesses in it. Like, you could take, like, you know, I know there's a lot of cases where people have these intentionally vulnerable applications or whatever. You might threat model that without looking for the vulnerabilities and then look behind the scenes to see what vulnerabilities were there to see if your threat modeling caught that those weaknesses could happen. Mm-hmm.

I think, Anton, this problem has a lot in common with other problems in security where it's really hard to know if you got it right, but you sure as heck can know you got it wrong when you got popped in some way you didn't anticipate.

Yeah. Many years ago, I had an old and wise mentor who always brought up the sharks with lasers in these discussions because he would say, well, you have an app and you're threat modeling and you seem to have completed this exhaustive process. But what about sharks with lasers? If they show up, are you going to be like, okay? And people would kind of, in a near-religious way, they would get enlightened. Like, oh, well, and then some other useful idea would come to their minds. I don't know. Somehow I'm stuck with sharks and lasers stage of TM.

That's an interesting point because it could sort of go either way. It could trigger useful things, or then someone goes and actually builds protections from sharks and lasers that we don't need.

Well, excellent point.

There is only one context I can think of in Google's security journey where sharks with lasers are very relevant. And that's a long story about a data center in Australia. But Meder, thank you so much for joining us today. This was truly a delight.

Yeah, thanks for having me. It was my pleasure. It was nice to meet you both.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Secure ready. Secure ready.