EP91 "Hacking Google", Op Aurora and Insider Threat at Google

Security.

Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a Reformed Analyst and Senior Staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we are talking to one of the people that was there when Google was attacked with the Aurora attack way back now, 10, 11 years ago. That's pretty freaking special, isn't it?

It is. And moreover, Mike has shared some of the details both in the Hacking Google video series and on our episode. that to me are kind of interesting. Okay, understatement of the year prize interesting. But they're interesting as in like he was there dealing with stuff for the incident that kind of changed the, maybe not the fate of the planet, but it had a massive scale, like society-wide. I kind of joke that it's planet-wide.

It is.

It's kind of a big deal.

Yeah, yeah, yeah.

It was a big deal.

Hey, what's the Hacking Google video series?

Oh, this is the interesting thing that got launched earlier this month that has video interviews with a whole bunch of Googlers dealing with security. It's almost like any description— well, no description would do it justice. You really need to watch them. They're not that long, but fun.

So people can find those on YouTube and learn more there, yeah?

Exactly.

Awesome. Well, listeners, I really recommend that you check out the Hacking Google video series. It's a pretty special look at how Google keeps people's data safe. With that, let's turn it over to today's guest, who helps keep everybody's data safe here at Google. Delighted to introduce today's guest, Mike Sinnot, Security Engineering Director for Detection and Response here at Google Cloud. Or Google overall, really. You were recently featured in our Hacking Google videos. We're really excited about these. Awesome content. Could you share a bit about this effort and the role you played in it?

Sure, Tim. Thank you and Anton for asking me to join as well. So yeah, I started life at Google as a systems administrator and specialized in Active Directory.

Wow.

Oh no, no.

Active— did you just say Active Directory on the air?

At Google?

I did say Active Directory. Yes, yes. This was a long time ago in a galaxy far, far away. So I was one of the domain admins for the AD, and ironically, about 6 months before the incident started, we had just completed renaming the entire Active Directory domain, which at the time I think had been like the largest RENDOM ever. So I was sitting on my couch and I got a ping from one of the security engineers saying, hey, I need domain admin for the duration of the security incident. And my first question was, what incident? And they said, oh, You don't know yet. Can you join this call? So I started joining, I joined the call and, you know, they started asking a bunch of Windows questions and then finally decided, can you just come out here?

Wow.

So I hopped on the plane, literally like first flight out of Newark the next day, because I'm based out of New York. And from that point, it's a big blur, but a lot of the kind of early work we did was reconfiguring the domain, implementing additional security controls. Making decisions on who had access, who didn't. Most of your Windows-y stuff that everybody kind of cringes at now these days. But as it progressed, it turned into more smart hands because I wasn't a security engineer at that time. I was just, not just a systems administrator, but I was a systems admin. So it went from ripping hard drives out of machines all over campus, which I do kind of talk about in the—

Which is totally a security capability.

Yeah, yeah. And I talk about it pretty much in depth in the video as well, but driving around campus just in the middle of the night, pulling hard drives, and then up to and including deploying the first kind of binary allowlisting software.

Wow.

On the infrastructure, because we realized that, you know, there was software out there we weren't sure about. So Sergey, Heather, and a number of people made the call to start pushing it to certain sites so we could lock down the machines. Which ironically, once I got home a couple of weeks later, we were going through year-end close and I had to reverse some of it because there was an issue with SIFs and it almost broke year-end close. So it was just a whole funny chain of events almost. But yeah, it was a lot of different tasks, all mostly related to Windows, but then just kind of things as they came up. We were grabbing everyone and everyone we could for help. So it was us, it was FieldTechs. Obviously Tim was there because Tim was also on the Windows team with me. Him and I were partners in crime in Windows and Active Directory.

Wow.

Yeah. So it was a very, very, very interesting 4 to 5 weeks of work just during the incident and all the follow-up work that came along with it.

And this was 2010, not 2009, right?

No, it was 2009 into 2010.

2009.

Yeah.

This is December.

Because I left home in 2009 and I got home in 2010.

Oh no.

So it was over Christmas. It was over Christmas and I remember having to call my daughters and be like, okay, Santa's coming via FedEx this year. And I shipped all of their gifts from Mountain View to their mom's house so that, you know, they had gifts from dad and everything else and called them on Christmas and all of that. But it was a, yeah, it was several weeks. And like I said, it was a blur. Like thinking back on it, every time I think about it, I remember something new because it was also, you know, 12 years ago.

A long time now.

Yeah.

But it's kind of one of those incidents that's the— I mean, it has a planet-scale impact. I mean, maybe I'm self— well, Google-aggrandizing a little bit, but the incident had kind of a planetary impact.

Yes. Yes.

I'm not overinflating it. It had technical impact. It had geopolitical impact. It has just political impact. You were involved with something that kind of changed the planet a little bit.

Yeah, especially within Google too. It was a very defining moment for us.

Yes.

Right. It was almost the end of innocence in a way. Not that we didn't have a well-defined security apparatus beforehand, but it was definitely an end of kind of almost a naivety in how we approached this, what our threats really were.

Mm-hmm.

So yeah, it had global impact. It had very local impact, but globally, yeah, it spurred the discussion. I think Secretary Clinton at that point had brought it up. In one of her talks, there were ripple effects felt throughout the world because it wasn't just Google that was affected as well. It was a number of companies.

I mean, I was a political science student in university at the time. We talked about it in class as a political event at that level.

Yeah. No, that's just like, we should probably all reflect about it for a second because this is not just like, hey, I had this complicated IR engagement back in 2010.

No, this was definitely not that. This was much more than that. And I think it woke everybody up in many cases across the industry to what was possible. Up until then, I think there was a lot of theoreticals, things you heard about. I mean, there was, there's definitely things that happened before that, but this was probably one of the most well-publicized, well-documented, and well-known cases of, you know, this large of a breach.

So to start switching gears to modern times, I just wanted to ask, you're dealing with other types of issues nowadays. And I hear that insider threat is back in the news, at least in some regards. So what's your current role in detection response, and does it touch on insiders, and in what way? Sure.

It absolutely does touch on insiders. I lead our insider threat detection organization. So we are the ones who detect and respond to insider events at Google, right? So we're looking for access abuse, exfiltration, of data across all of Google, really Alphabet in many cases, because we also engage with the, you know, the member companies. So the bets. So that's my role. I lead the detection folks, but I partner with all of the engineering teams, all of the other— there's insider risk engineering teams that focus on how do we minimize and put preventative controls in place as well. So it's a lot, a lot of cross-functional work. But, you know, our main focus is, you know, detecting and responding to potential insider events at Google, right? Because a lot of cases, things that look like insider just end up being minor mistakes, people making workflow mistakes, doing the right thing in the wrong way or the wrong thing in the right way. So yeah, it's a very interesting job. It's, it's never the same thing twice, but it can be, you know, it can be hairy. It can be very hairy.

So you mentioned mistakes and you mentioned the well-meaning mistakes. And that's been my, uh, in my engagement with insider threat back in my analyst days. This was like a painful question every client asked, including us at Gartner. Do you deal mostly with like the big evil insiders, like implanted spies, you know, horrible scenarios you can imagine? Or do you deal with people who are kind of well-meaning and not malicious, maybe make mistakes, maybe like you are literally in the trenches with this in this battle. So I don't want to turn it as which one keeps you up at night, but like, what's your take on this balance between well-meaning insiders?

Yeah.

and just legit evil insiders? I don't know. That's not a technical term, legit evil, right?

Am I looking— No, but it could be. We deal with both and we see definitely a higher percentage of more well-meaning mistakes, people doing the wrong thing but just unintentionally causing a problem. But I think in terms of what's keeping me up at night, to steal that from you, is the malicious insiders, because they really are out to cause damage, whether it's foreign espionage or people looking to take down Google or, you know, kind of other malicious actions. I think, you know, volumetrically, we see a lot more accidents. People click the wrong thing, shared the wrong doc to their personal Gmail address instead of their, their Google Gmail address because they're exactly the same and the dialog had both and they clicked the wrong button. So volumetrically, the risk is— there's more risk there. But I think in terms of impact of risk, malicious insiders, there's kind of a larger impact once they get a foothold.

And they do come up in many kind of classic— I don't know if there's a book of classic security leader nightmares, but malicious, technically skilled insider, in my mind, was always high, close to the top of the list. Imagine somebody with essentially the tech skills of a top-tier attacker who is also an insider. I'm probably going to lose sleep this night, frankly. I just said it, and I'm not even responsible for it.

We don't have to remember that or imagine it. It happened, and Google was relevant in that one too. Remember SSL added and removed here? That's exactly that case, Anton. Yeah.

I mean, and the attackers, in terms of an insider, have foreknowledge of our systems. They know how the systems work. They know where the holes are.

They know where the detection is and how to get around it.

Eh, not always, right? Not always.

Hopefully not always. I mean, it's a balance between being able to detect it, but I would much prefer we prevent it in the first place.

Yes.

Right. Training the users, because again, the unintentional pieces, training the users is a big part of this, but also putting in strong controls up front so that it doesn't have to get to detection. I'd much prefer that because once we get involved, in some cases the horse has left the barn. Not always, but in some cases the thing has already happened. We can do attribution for it. You know, we saw that it happened, we can respond to it, we can remediate what happened, but it still happened. So that's why, again, I focus on the partnerships aspects of how do we prevent this upfront so that we're not involved on the backend. And if we are, we are. But, you know, and of course that happens plenty of times a day, but strong prevention and then strong detection. complement each other very well.

Yes, they do.

So we're going to run an ad for Zero Trust something right now, right, team? Okay, running an ad for Zero Trust.

This episode brought to you by Identity Aware Proxy. Learn more. No, so, so I think what's really interesting though is, you know, so much of our job is about trust at the end of the day. You know, people trust Google with their most personal, intimate things— their, their photos, their correspondence, their searches that 2 in the morning. I imagine a lot of what's the value of insider threat detection is really preserving user trust. How important is that to your mission and what you think about in the day-to-day?

It's core to our mission. You know, like you said, users, our users trust, and even our employees in many cases, right? A lot of the data internally is just as sensitive as the data externally. We have people's information. They trust us with that data. Everything is built on trust. So having strong controls, having detect— you know, detection and response in some cases is the, you know, the last line, the last stop in that journey, where we're the last line of defense there. If somebody slips through the controls, you know, it's up to us to detect it, then respond to it, mitigate as appropriate, remediate as appropriate, and then take the findings from that back to the engineering teams or to the affected teams. to try to prevent it from happening again, right? To learn from our mistakes. So yes, it's core to our mission. It's the kind of whole— one of the whole reasons we exist is user trust, right? User data at Google is sacrosanct, the crown jewel of our crown jewels, right? It is given the type of data, the data that's stored, the impact it can have on real people out there in the world. It's something, you know, our entire organization exists to protect that data. Intellectual property, obviously, is another aspect of this, but user data is what everybody— and it's not just detection and response, it's not just security. Googlers in general really respect the trust our users hold in us for their data.

So one fun thing to add to this, and this is kind of— I mean, I'd rather this kind of sink in with the audience, right? Because It's not IP. It's not like some kind of piece of code, how to do something incredible technically. It's kind of important, but it's secondary after user data. This is sort of a good reminder and good thing to kind of stick on the wall. And I think it got to me too. Like, I feel that, that it's user data first, IP second.

When you say user data first, IP second, like you feel bad how? I'm just curious. No, no, no.

I don't feel bad. It just feels like it's something that I need to kind of internalize and understand fully. I agree with that, of course, but it's interesting that we think of it explicitly. It's crown jewels. The crown jewels is user data. And all other crown jewels are probably hugely important, just not as important as this.

Yeah. I mean, it takes a second to digest, but if you had to stack rank them, just the impact that loss of user data can have on people, It, to me, trumps everything else. That doesn't minimize the impact of loss of intellectual property, the state secrets, the robotics, the artificial intelligence, you name it. But user data, people's data, given the effect it can have in real life, as we've seen in other instances outside of Google, on people, is the core of trust. the core of, you know, of core of what we do, with then intellectual property as well. It's not like we don't detect exfiltration of intellectual property and follow up on it, but, you know, order of precedence, it's user data, intellectual property.

Yeah, that actually makes sense. And I think that a lot of listeners probably would say, I'm sure Michael has some kind of super powerful technology, obviously using real AI or something, that detects threats. So Question about the tech stack. It doesn't have to be super powerful, but I'm kind of curious, what can you share about the tech stack that your team uses for this? And I've seen some references in public about this, so what can you share?

And I'll lead the witness because I work closely with your tech stack. One of my favorite things is BrainAuth. What can you tell us about that?

BrainAuth is actually an umbrella program that has several projects underneath it. So I think what you're probably most familiar with is the facade model.

Yeah.

Yeah, it's basically— which basically does contextual anomaly detection and looks for actions on supported systems that are unlikely for a given user-system pair.

Hmm.

So, you know, Timothy did this thing on this system, but has never done it before. And given role, etc., it's unlikely. And we'll then surface that to us. So we ingest that data, We do automated analysis on it, further enrich it, and then review the output and do any additional manual follow-up that has to happen on it. There are a number of other models, but that's one of the biggest, and that's one of kind of the first ML models that we started with. The BrainAuth team could better explain it in detail than I can, and we could probably have an entire episode about that because it's super interesting.

Yes, we could.

Because I think they presented it in 2018 at a Microsoft colloquium on on machine learning. I was actually re-reviewing the presentation that they did this morning about it.

Oh, nice.

But, uh, you know, that's one of the tools in our toolbox. There's also a lot of additional detection that we write purpose-built for various workflows, various types of data that augment it. So it's, it's one very, very, very good signal that we have, but it's also not the only thing we rely on. Right.

So listeners, if you'd like to learn more about that one, let us know in the comments and on the LinkedIn and on the Twitter, and we'll, uh, see about getting those folks on the show as well. Mike, one of the things you've referenced there is automation on top of those signals. Could you share a little more about the role of automation? Because we love talking about automation on the show as well.

Sure. Automation again. So SST, which I know you talked to Tim a couple of weeks ago, and my team ITF.

What are SST and ITF real quick for the listeners at home?

The Security Surveillance Team and my org is the Insider Task Force, which actually was born out of the SST.

Mm-hmm.

So it started in SST and then grew to a size where it became its own organization.

Mm-hmm.

Because we run, we run 24/7, my team. But we share a good 90% of our tech stack, right?

Mm-hmm.

We diverge in certain areas because of just the threat models. But so data gathering, analysis, and we, we even infer initial priority and we're working towards automated mitigations. In some cases.

Wow.

So a lot of it right now, what we're testing for automated mitigations is kind of the lower severity events where we would email. We have an automated email system that will query the user and say, hey, we saw this happened, can you tell us why? And there's a bunch of, bunch of checkboxes that go, you know, I did, you know, I was doing this, or I— this was business justified, etc. So we're starting there. But we're looking at even automating the mitigating, the mitigating actions.

Oh, wow.

So that we don't have to take that action manually. But it's a very— we're very slow and very careful there because it could, you know, could cause chaos if we, if we tried to go too fast and end up causing more problems than we solve.

That makes sense. Yeah.

And then, you know, from the ingestion standpoint, you know, logs ingestion, we don't have to go out and generally gather logs there. Pulled into our pipeline, presented in front of us, and then the automated analysis is once a signal fires, it'll get pulled into a hunt. Depending on what that hunt does, it presents an automated— here's all the things you would have done if you were doing it manually— screen to them with hints on what this could be, and in some cases a conclusion on what it could be, and then they will do follow-up actions based on that. There's judgment that still needs to be applied in many of these cases. So they'll take a look at the data, do any additional follow-ups, additional queries, additional looks, and then either escalate the case or close it as, oh, this was a business justified action. They're sharing data with a partner that has an NDA, et cetera. Or we really can't explain this, so we need to move it to the next step.

You said the word hunt. For listeners at home, that really means like a playbook that ran, right?

Yeah, it's essentially an automated playbook. We write hypotheses for various threat models to say, in this case, if this happened, we would investigate as such. And it steps through that automatically and then presents an analyst with almost a fully formed set of data rather than them having to go out and pull the data manually where it would take much, much longer and probably get really tedious for them. Whereas now they can look at it and only pull any additional data they need to support what they're seeing.

And that's one of the things I love about that is exactly that tedious. It's a total TDM reducer or toil reducer, which is huge.

And it helps eliminate analyst fatigue as well. Yeah. So, you know, they're, they know what to expect, they know what they're going to get. And in some ways, the repetition helps because they know what patterns to look for within the tickets. It's a two-sided coin there because we need to be careful of the same thing from a fatigue standpoint, that they're not just looking there, that people are applying some critical thinking as well to say, but there's this thing and it's kind of strange. What happened here?

But notice, by the way, it's a good pointer to the audience is that some of the automation is what you'd expect in a normal outside threat-facing team, like a classic SOC model, but some of them are very different. And so this is kind of a useful reminder to everybody that just saying, hey, we automate, and they point at some random playbook in a SOAR, like, it may not be the automation you need for some of the insider issues. This is not just classic SOAR playbooks. A lot of this is kind of engagement with a human type of automation.

Yes.

Yeah. So we're running close to time, and we have the classic final question, namely— well, it's kind of split into 2. Any recommended reading on the topic? And one tip to improve insider programs, insider threat programs.

Sure, I have a bunch of recommended reading, and from a tip to improve insider programs, I actually have 3, but I'll start with the highest order. I will take 3. Okay, 3. Number one is don't try to boil the ocean, especially for insider programs that are just getting off the ground. Pick your biggest risk items, start there, and that can vary based on the company, Or the industry they're in. And if you don't have a risk register, now is the time to start building one because you need to look at your risk and stack rank and say, where are we going to expend our resources to start looking, detecting, preventing insider risk? So number 2 is building partnerships across your organization. From an insider risk standpoint, legal and HR are very obvious ones. They always have to be involved. But also work with your product areas or business units or whatever your organization calls them. and your dev teams to look at their very specific risk and create that feedback loop with them so that when you start seeing trends in incidents, you can communicate it back to them and say, we see a trend here, what's going on? How can we fix it? And then number 3 is understanding company culture. Every company's culture can be very different. Google, Google's a very different place than a lot of organizations. So we have a very open culture. So understanding how to fit your insider program in with company culture is extremely important for acceptance of the insider risk aspects. And also, you know, knowing what controls you should implement, what type of detection strategies you should implement, et cetera. And then for recommended reading, this obviously the Secure Reliable Systems book, page 24 has a very, very good matrix of how we consider Insiders: First Party, Third Party, and Associated, but also the CERT Guide to Insider Threats, Common Sense Guide to Mitigating Insider Threats, which is, uh, by the Software Engineering Institute at Carnegie Mellon because they sponsor the Open Source Insider Threat Sharing Group, which a number of companies across many industries are a member. There's another book called Insider Threats from Cornell, so it's the Cornell Studies in Security Affairs. And then for, from the IP perspective, there's a good report by the NCSC called Foreign Espionage in Cyberspace that breaks out a lot of the critical technologies, what foreign actors would be looking for insiders to take, et cetera. I actually have links to them, so I'll send them all to you. But yeah, I could go on this subject for quite a while, but that's my top 5.

Well, Mike, thank you so much for joining us today. Listeners, check out the Hacking Google video series on YouTube. And we look forward to seeing you again soon, Mike.

Thank you. Look forward to seeing you too.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.