Modern Threat Detection at Google

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks so much for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed, tamed, and captured former analyst and a member of the cloud security team Here at Google. You can find this podcast wherever podcasts are distributed and at our website. If you like our content, please consider hitting that subscribe button on your platform of choice. You can follow the show and argue with us, your hosts, on Twitter as well at twitter.com/cloudsecpodcast. Our episode today, I'm super stoked about, Anton, because we're talking about my favorite thing, which is threat detection. And knowing a thing or two about our guest, I think it's going to have your favorite thing too, which last time I checked is logs.

Lately, I've been not just in love with logs, but also with threat detection and response as well. Also, today's episode will reveal some of the secrets— secrets— about how Google does it. And Google does it really well, as you know.

I like to think that Google does it pretty well. And for the legal team, when you're listening, we actually made sure there were no secrets, despite what Anton said. So lawyers, pay attention to that. Listeners, lots of secrets ahead.

Exactly. And to me, some of the toughest things in threat detection is really how the threat knowledge becomes detections. A lot of organizations I've dealt with in my Gartner days and since that time basically know something about the threats, and they sort of know they want to detect them eventually, but this bridge they cannot cross. They cannot get from, here's the threat that may do this, and I want it to be in my SIEM or in my EDR, in my other product. So today, a lot of the stuff that Julien would cover actually covers crossing that bridge.

And, you know, it's funny, that bridge, that translation of, I'm worried about outcome X, to, I have rule Y that encodes a capturing of it, that's the core of what I do and what my team's doing. It's a fascinatingly fun problem. I just love it. So perhaps with that, let me introduce our guest today, Julien Vehent, a security engineering manager here in Google's Detection and Response team. Delighted to have you on the show today. And Anton, I think the first question is for you.

So for the first question, I've wanted to focus on some things that are very fun for me, detecting modern threats, kind of top-tier threats, interesting technical threats in modern environments. What is modern environments? It may be cloud. It may be things the way Google built them. Just kind of use your own interpretation. You want to be as cloudy as you want, otherwise focused elsewhere as you want.

Thank you for having me. Happy to be here. I think to dive right in, the main thing I want to highlight is that there is a lot of similarity between detecting threats traditional on-prem environments and cloudy environments, right? Where we're going to focus on as far as differences is really on the control plane layer, the identity and access management layer that doesn't typically exist in an on-prem environment, where in cloud it is absolutely the core of everything that we do. So those identities, those service accounts, they have keys. Those keys tend to leak. When they leak, they can get misused to perform bad actions in the environment, to use resources of the account to mine cryptocurrencies and so on. And that's really where the bulk of a cloud detection team is going to focus its energy, really.

But wait, wait, wait, wait. So this sounds like regular threats, though. Like, I think we asked for modern threats in modern environments, and you gave us, like, regular threats in modern environments. So you gave us only half of the answer. So what about something more interesting?

He mentioned IAM. You don't have IAM on your on-premises data center. Give the guy a break. It's his first question on the show.

Oh, okay, fine.

All right, let's go with the quantum threats. Let's go with the quantum threats. You probably have some IAM on your on-prem infrastructure actually, because a lot of even traditional data center type infrastructure will have a hypervisor layer and will have a provisioning layer. And that's basically a control plane, right? What cloud does is just boost that up to 1,000 and give it a lot more power and flexibility and becomes a more central part of the infrastructure management layer. But with that said, as far as quantum threats go in cloud, we in fact have a lot of the same old stuff that is just done a little bit differently. Lateral movement is done by stealing credentials and moving from one resource to the next. And what we see also is that the attack surface is a lot smaller because the cloud effectively exposes a lot less of the infrastructure, of the traditional infrastructure, to customers and to attackers. So we in fact start with a status that is a lot more secure by default. That's the main point of cloud here. And the work of the detection team can then be focused on more advanced threats instead of really just focusing on the basics of an IoC going through an IDS that would consume the entire resources of the security team.

Or closing firewall ports, as they were doing in the sort of 1990s, I guess. Interesting. By the way, smaller attack surface area is a useful reminder for the audience. Cloud means Smaller attack surface area. This is a good reminder to everybody, and thank you for saying this.

Julian, this is kind of a fun question for me to ask since I know the answer 'cause we work together on it. How does Google turn your knowledge of threats into detection rules? What's the actual process for going from, hey, we're worried about X, to, hey, we're pretty confident we can catch when a bad guy tries to do X?

Yeah, this is very, very secret sauce that we never talk about anywhere, right? It's logs, right? A lot of it is logs. We consume a lot of logs from a lot of different places. And logs are really the blood of a detection infrastructure. The Cloud Audit logs coming out of GCP will contain incredibly useful information to a detection team. And then you mix those logs with a fairly sophisticated detection pipeline that leverages threat intelligence that is often proprietary to an organization, often public as well. We mix the two together and turn those into signals, into findings that a detection team will review and escalate and remediate as needed. But it really all starts with—

Yeah.

Good quality logs.

But when we had this question in mind, we kind of thought about the process of turning the knowledge about a threat, like, you know, the attacker might do X, but then what you have in your hand is, say, Cloud Audit log, DNS logs, other type stuff. So how would you connect what you know about the threat and what they may do to the signals you have? To me, this is kind of the juicy part of the question.

This is where the detection engine comes into play, right? And we really want to focus on applying the intelligence that we have to the logs that are passing through in the most efficient possible way. This is often something that security teams will build internally, and they can at times rely on existing products. I mean, GCP has very good quality security products for this, ETD, Chronicle, et cetera. Splunk has been on the market for a very long time, and there are other SIEMs out there that exist. The focus here really is that a security detection team will essentially engineer detection that consumes those logs, apply intelligence, turn them into usable signals. How we do that is actually very complex to answer because it entirely depends on the type of detection we're trying to build. But it's code.

You have different engines for doing this in, right?

That's right.

Some of which are pretty cool.

A lot of this will be essentially starting with a very large pipeline of logs. You have a lot of logs going through a streaming engine. And you want to mix the use of, for example, large databases. BigQuery is excellent for this. You can run queries across an entire dataset of logs that goes back many, many months fairly easily at a fairly reasonable cost, right, without having to worry about the storage and the capacity you need to run this. Other engines would be more real-time. For example, you may want to inspect those logs as they flow through a queue and then throw them away because you don't care about retention. You just care about the real-time signal. Those signals, when they are created, when you find something that's interesting you want to surface, it's good to route them to a queue that can be triaged by a human being. A lot of organizations will use bug trackers for this. You can use SIEM, you can use existing tools on the market. The point is there needs to be a record of a given signal being entered into the system, triaged by a human being, and possibly reviewed later on. And the point of this is that a signal may not be a true positive at a point in time, but later may become a true positive when new intelligence comes to light.

That's another very useful reminder. You may rate something false or perhaps inconsequential, but like 2 hours, 2 days, 2 months later, it may come to light that it was a big deal, right? Like roughly, hopefully not 2 months. That sounds like bad news to everybody, but that's what you're talking about when the seemingly false unrelated signal would be rated important kind of in the past.

That's right.

The other place where we see organizations wanting to keep things hanging around is for the compliance case of being able to report on all the possible things, even the things that were, you know, risk accepted or controlled by some other control.

Must you really bring the C-word in the conversation, Tim?

I'm sorry. I'm sorry. I should put a dollar in the swear jar for that.

Can you go through one example of creating a detection for a new threat? Like, you know, you can pick something hypothetical or something that you've really done. Doesn't matter if it's made up, what I'm trying to say. Start maybe from threat modeling all the way to writing a rule. Like, how would you go through that?

Right. And I'm glad you mentioned threat modeling because this is really where the work starts. It's impossible to create good detection without truly understanding what we are chasing, what we're hunting for. And this is where good threats and dictionaries of threats exist that can be used to build those threat models are really useful. So the first step is for a security team to actually talk to the folks in charge of a given product or project and perform a threat model with them, understand what they want to build detection for. Some threats may be generic and inherent to the infrastructure itself, but some threats may be specific to a given product. So for example, if you have a service that enables users to create an account and log into your system, right away you may be concerned about password stuffing attacks against that system because an attacker may take a leak of username and password and try it against your service. And that's really the first step. It tells you what threats you need to build detection for. The next step is to really review the state of the art of attacks that may exist against this particular product or class of software. If you are, again, building a REST API, for example, then there is well-documented threat models for REST APIs that give you hints as to what type of detection you want to build for that service. The next step is to essentially review your logs. And if you don't have good enough logs, to work with software engineers on issuing better logs for the detection efforts. I think this is a really important point that is often missed by a lot of teams. You can't, as a security team, really do this in isolation. You have to work with the engineers in charge of a system in order to get the good logs, to get the right type of information surfaced into those logs, and get those logs flowing into a detection pipeline where you can then apply threat intelligence to the detection effort and create those security signals.

I relate to that a lot. I spend a lot of time talking to teams about the logs I wish we had.

This is so sad. I'm going to cry. This is Google. We can make whatever logs we want. We are rich, right?

It's not that we can't afford the logs, Anton. It's that the logs simply aren't written because until security team came along, nobody was like, oh yes, I want to log this obscure behavior.

Well, there's that.

I don't think I've ever done a postmortem— I've been doing this for a long time before joining Google and been through a lot of incidents, and I don't remember a single postmortem where we didn't at some point mention, and we wish we had those logs. It always comes up at some point that you want to have that one specific log that will essentially catch the action an attacker performed, and it never exists.

Right.

But it's a constant process of essentially continual improvement to identify the logs that you're missing and work with the team to get those logs issued.

Let's talk about that process of continual improvement because that's an interesting one because we on the podcast, we don't just want to talk about technology and people. We want to talk about process and process improvement. And how do you process improve? How do you test your detection rules and content? How do you know if it's working?

So here at Google, we're lucky enough to have fantastic red teams that can help us essentially stress test our infrastructure. And work with them to verify that our detections are working. A lot of organizations apply that model of having the red teams and the blue team kind of work together to test the detection and continuously improve the quality of the security infrastructure. Failing that, I'm a big fan of just hiring security firms to come in for a couple of weeks and test the security of a specific service or part of the infrastructure, right? It's a great thing to do to bring that external knowledge to maybe a smaller organization and have them test out the security. It removes all sorts of assumptions may have been built internally around the security of a component. There are also automated tools that exist on the market to generate essentially fake attacks. Like, we will do a unit test or an integration test that will essentially tell us that a signal has been created for a prerecorded malicious behavior. And those are useful as good software engineering practices because at the end of the day, detection engineering is software engineering. And all of those practices that are good for regular software, they're good for I like that.

Did you hear that, Anton? Detection engineering is software engineering.

That's something I think I've said in one of my blogs, yes, but I see it come to life here. Yeah, yeah, but you're right. I mean, that's the advice that I've seen people kind of take into practice, and I've seen people who do not want to take this advice into practice. Like, I've met clients who say, I just want to buy a tool, push a couple buttons, and have my detections running right there. I don't want to write code. So I've seen people who kind of object to detection engineering, detection-as-code approach, and they would argue with us here and say, no, I think detection is pushing buttons. It's not writing code. So we know better, but there are enough people who don't, and it's kind of a useful frame about the world that's bigger than Google.

And one last point I will add to how to test your detection, and one that a lot of organizations have adopted, is to essentially get hacked, and that's a great way to test if your detection is working. There's a saying in the industry to never waste a good incident, and I think the point behind this that is very, very true is that Getting hacked not only stresses your entire detection and response organization but also your entire company and will force everyone at every level to essentially step up their game. And that's a good opportunity for organizations to apply some introspection and identify areas where they need to invest and improve. And I think it's a valid way to improve your detection strategy.

That makes sense. And so I want to quickly journey from the process to the third pillar, namely people. So one of the things I've encountered recently when talking to organizations that are building security operation centers is that some of them still have walls between people who write detections and people who triage alerts. So to me, this is a very dysfunctional thing to do, yet it's still very popular. So I hear, you know, through the grapevine that at Google, we never do that. We kind of have the same people who write detection, write detection logic, write detection code, are the same people who respond to signals. So would you care to kind of explain how it works, why it works well? Because I've had a lot of arguments with people who really don't want to do that.

It's a very simple thing. If you're not in charge of engineering an alert, do you really care if it fires off at 2 AM in the morning?

Hmm.

It's not your responsibility. And this very simple rule, in fact, creates a lot of tensions in organizations where the responsibility for writing alerts and triaging them is separated. Alerts are noisy. No matter how much effort you put into writing detection, there is going to be a point in time where some detection fires off continuously way above a reasonable threshold, is triageable. And it is important to take those very seriously because alert fatigue is a real thing. And it will stress out an entire response team. Having the same team essentially write the detection and triage the alerts is just a better dynamic to make sure that alert fatigue remains under control, and that everyone who participates in the effort is accountable for the quality of the detection engineering.

Yep, that does make sense to me. And again, it takes a bit of a journey to build it that way, though. I've seen people who You cannot do it day one. I mean, Google evolved to that, correct? We lived and then eventually we got to this good state, right? So it takes work.

Yeah, I'm not particularly familiar with the history of how this was brought up. There's certainly a very high level of maturity here that I've observed, and I've observed it in other organizations as well. At some point, someone does it wrong and they learn their lessons from it and they correct their strategy pretty quickly.

One thing about Google and where we come from is you said one of the best ways to test your detection is to get owned. That's what happened to Google a long time ago with the Aurora incident that we went very public about. I mean, a lot of what we see on the teams that you and I work with came out of that incident. And so I think really that the genesis of a lot of this is exactly in that kind of forged by fire scenario.

And again, it goes back to never waste a good incident, learn the lessons, and make sure you invest at the right time and with the right amount of effort to correct the kind of issues you're observing in the real world.

So to further go into that theme, it sounds like the operation similar to ours would call for somewhat different skills, not just people who can click buttons and kind of escalate alerts, but there's a lot more to software engineering. There's a lot more to agile and understanding hyperscale environments. So could you quickly go through a few top skills for a security analyst or a security engineer in that type of a modern environment?

I think it's becoming obvious that software engineering is at the core of every security discipline. We talked about this a little bit. It is very important to highlight that the myth of the security engineers who come out of, strictly speaking, like infrastructure management and can evolve in a modern cloud environment without learning programming and software skills doesn't exist anymore. Operating a cloud means writing code every day. And operating a detection infrastructure that protects a cloud means writing code every day. But it goes beyond that. It's also about adopting the practices of a software engineering team, about testing, about having, you know, well-documented code, about being able to quickly scale up the detection effort as the cloud itself scales up. So what we find is that security engineers who have a solid software engineering background are usually successful, right? And security engineers who lack the software engineering background, they tend to struggle a little more and they have to acquire that part. The rest of it hasn't really changed much. We're still looking for very curious and smart folks who are willing to keep asking questions, keep asking why until they get to that one assumption that was made that turns out to be a potential vulnerability. That kind of curious mind of not accepting the status quo is still a strong indicator of someone who will be successful in the industry. So I think it's really a mix. There isn't one really mold that fits the perfect security engineer. I think a good, healthy team will have people of different backgrounds who bring in expertise to the effort. And it's often useful to have 2 or 3 people look at the same series of log lines, look at the same activity, because they will draw different conclusions. And we want to capitalize on that. So a modern team is good at software engineering, is very diverse, is very curious, And is constantly assuming that they don't know enough.

I like that last one. One of the things we've seen be useful for some of our users, and I know we do it a little bit here at Google, and I actually want to do a whole other episode on it, so I want to just quickly touch on it now. Could you speak a little bit about the value of an asset inventory or risk register for how we're thinking about our cloud assets?

Yeah, this is indeed a vast topic. You can't detect what you don't know about. And if you don't know about an entire class of assets inside of your infrastructure, you won't get any detection around those assets. And potentially, they become a perfect entry point for an attacker. That's the first part. And so in order to protect the entire infrastructure— and by the way, cloud has dramatically changed the game in that space because with cloud, you automatically have an inventory of everything. You can programmatically query your infrastructure. You have asset inventories that record the history of asset creation for long periods of time. And this is a tremendous advantage in detection efforts. So that's the first part. But when I started this, we didn't have those. We had to do incident response without having any sort of asset inventory, without knowing the history. And every single time we were wondering if a host existed at a point in time, when a host was created or shut down, if the host could have been part of the lateral movement of an attacker, et cetera, et cetera. That's really what is key about asset inventory is that it answers critical questions that we need to answer when detecting and responding to threats. The risk register is a little bit different. The risk register is about focusing detection on the parts of the cloud infrastructure that are the most critical. And in large organizations, there are always layers of infrastructure that are at different levels of risk. And it's impossible to focus detection efforts on the entire infrastructure, including every single test environment, every single defunct infrastructure, et cetera, et cetera. And the risk register will allow the detection team to say, the signal is coming in. This is for a test asset that doesn't contain any critical data, so we can deprioritize it, look at it later. But that other signal is coming in, and that one is related to a critical part of, for example, the accounting infrastructure, and we want to escalate it to an analyst right away.

So what's the connection between that and detection and response, real quick? Is this prioritizing how we triage? What's the connection?

Responding is mostly about responding as fast as possible. In order to respond as fast as possible, you need to know how critical an asset is and how quickly you need to bring in a human being for response.

That does make sense, yes. So as we are starting to kind of wrap up the story here, we want to hit you with 2 traditional questions that we like to close with. Please name one practical thing the audience can do to improve detection and response in their environments, cloud or not. Just like, Your favorite tip on DNR?

Go talk to your developers and ask them what they are worried about in their applications.

I like that. That ties in with what Alyssa was saying about doing threat modeling when you're writing your stories. Our other traditional closing question is, do you have further recommended reading for the listeners so that we don't leave them empty-handed today?

Well, there are a couple of excellent books from Google that I think you may have mentioned already, the SRE book and the security engineering book. You can read mine. It's pretty good as well, Securing DevOps.

I honestly did not know that, so we'll absolutely feature it on the podcast page. Thank you.

That's awesome. I didn't know we were doing a book show episode today. This is great.

This is great.

And yeah, I mean, that would already set you for 1,500 pages, so I think you're pretty good with those 3.

I think any more than that would be an act of cruelty for our listeners. Julien, thank you so much for joining us today. Really a pleasure to have you on the show. I know this is a wide-ranging conversation, and listeners, thank you for bearing with us as we hopped all over the place when it comes to the question of detection and response in the cloud.

Thank you for having me.

And folks, we're at time with that. Thank you so much for joining us today. Like I said at the beginning, you can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever you get your podcasts. Please subscribe so you don't miss our future episodes. Check us out at our website and you can follow us on Twitter as well. twitter.com/TheCloudSecPodcasts. Your hosts are @Anton_Chuvakin. And myself at underscore Tim Peacock. Tweet at us, email us, argue with us. If we like or hate what we hear, we might even invite you on the next episode. And with that, we'll see you on the next Cloud Security Podcast.

Security.