EP158 Ghostbusters for the Cloud: Who You Gonna Call for Cloud Forensics

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button in your podcasting app of choice. You can follow the show, argue with your hosts, our other guests, and the rest of the listeners on LinkedIn. Anton, this is a fantastically fun look at how Google does What exactly?

Forensics, but particularly cloud forensics.

Cloud forensics. That's a great topic. How do you do cloud forensics? Surely, you can't go pull the hard drive out of the data center.

We did ask that very question of our guest. But to me, the moment of, like, my mind is almost blown was the moment when we talked about who are you going to call, Ghostbusters. But then the question arises, who would Ghostbusters call for help?

Right.

That is literally Really hard to answer because you can sing the song many times, who you gonna call, Ghostbusters, but then you think, wait, who would they call? And then you're like, oh my God.

That's right. So maybe without further ado, let's turn things over to not Peter Venkman, but today's guest. And with that, listeners, I'm delighted to introduce today's guest. Today we are joined by Jason Solomon, a security engineer here at Google. Jason, I won't give Although you may feel free to, your funny title. I want to thank you first for joining us. You've been at Google for a long time now doing forensics. Could you tell us about how you got pulled into doing incidents and why? How did you end up with that, that job?

Well, I started my career in computer forensics quite a while ago, and that was more in a litigation support kind of role, doing court cases, things like that. From there, I moved into a consultancy role. Doing incident response consultancy. And that's kind of where I got into the security space. And then from there, I moved here into Google, where I've been here for about 4 years now, a bit over 4 years on the security response team here at Google. When we get pulled into incidents, the investigation varies greatly case by case, but the goal is usually the same. We ultimately want to determine whether Google or Alphabet or our users were impacted by a potential security event. And those events could be anything from malware on a corp system, someone opening a phishing email, right through to unusual commands being executed or strange network traffic or unusual RPCs against the prod service. And that's kind of how we end up getting pulled into these things.

But wait a second. So given your background in traditional computer forensics, you are now dealing a lot with cloud instances, presumably, because I think it sounds like a compromised laptop, would be largely similar to what it was maybe not 10 years ago, but 5 and more years ago. But now we have to deal with cloud. How does it change? There's no obvious physical chain of custody. Like, you can't lock cloud in the basement. How does cloud change that?

Anton, do you solve problems by locking them in the basement?

I mean, pouring some concrete also helps. So I think it's both really should be used, not just one.

Oh, boy.

Defense in depth, you know.

Yeah, defense in depth of concrete above your enemies.

So I guess starting with our corp environment, when forensics is required, we would typically use the telemetry we have. So we do actual forensics a lot less than I have in previous environments because the amount of telemetry we have here at Google is insane and often makes forensics obsolete in terms of we don't need to go diving into the system to figure out everything that executed on a system as far back as you like. But we do still get cases where forensics is required, and then we most typically would acquire artifacts in a targeted manner using something like GRR, which is our endpoint agent, GRR Rapid Response.

The G in GRR stands for GRR, is that correct?

Yeah, it's a recursive acronym. I don't know why they did it that way, but yeah, sure. But it is kind of rare that we do need to do traditional forensics in terms of like collecting a disk And given the scale of our fleet, we have a tool that's open source called Gift Stick, which is basically a bootable USB Linux environment that we can kind of have someone on site plug in. And it basically takes an image of the disk and uploads it to cloud, to our cloud project for analysis. It's pretty rare that we have to actually ship a disk to ourselves and take traditional forensic image, but we do have forensic labs for that. Now, to get to your question, Cloud forensics is different, but the principles are very much the same. Basically, a lot of cloud forensics is just Linux forensics. Depending on the scale of what you're investigating in cloud, if it's a small incident involving maybe 1 or 2 systems, you can still do traditional forensics in terms of collecting a disk image or a snapshot in cloud. Cloud tools make it pretty easy to do that. The only issue with that is a lot of the time in cloud, you're dealing with larger environments like big Kubernetes clusters and things like that. Where that approach is not going to scale.

Because of humans or because of tech? Sorry for interrupting, but it won't scale because humans would break or the tech would break? What's the limit of scaling?

Yeah, if you look at the scope of a compromise, when you think about how Kubernetes can scale up to hundreds or thousands of nodes, we just don't have the manpower to collect all that data. Well, there is a tech element to being able to store all the data that you're collecting, but you just can't feasibly do a deep dive analysis on that many systems. We need to be a bit more targeted about what we're acquiring. And we also deploy our GRR agent to Kubernetes clusters and things like that inside a container. And then you can targeted way pull artifacts from nodes in that way. The artifacts that you collect live would then be preserved as evidence rather than the entire disk of a node. There's also a lot of valuable information in cloud logging.

Yeah, let's talk about that because that to me feels like what's super different about cloud, right? Like we don't have—

I think it's—

I want to interrupt Tim and I guess—

You want to interrupt Tim? You interrupted me, who interrupted Jason. Go for it.

Go for it. Yes, exactly. Because I feel like if we go to log analysis, we're going to depart in a more traditional, like, log searching, SIEM type of techniques, and we will depart from, like, the truly juicy part of cloud forensics.

So, this is where I disagree. I think there's something juicy here, which is that cloud forensics, you have a unified control plane, unified identity, and it's probably easier than ever, right?

Okay. Let's have you guys talk. I'm going to interrupt you again. Jason.

Yeah, so there's a wealth of information in cloud logging depending on how it's configured. You can sometimes find all of the logs from Kubernetes containers are also stored in cloud logging, so you don't even need to go to the node, for example, to figure out what was running on the system. It's just a matter of working out how long those logs are retained for and how to query them, basically. They can be quite complicated because they're all JSON log format and they differ from system to system, and it can be a task to understand what they all mean. But if you spend the time and learn what logging is available to you, it can make your job as a forensic analyst a lot more easy.

But actually, logging is hard, but going through forensic images to me is harder. So I almost feel like logging complexity is less annoying, less irritating than going through images. So it's kind of good news then, right?

Yeah, I certainly find it a lot easier to just query some logs than to go and dig through an operating system finding artifacts.

Yeah, that makes sense. So when it comes to dealing with this new world, what are the kind of tools you're using beyond GRR? What else is in the tool belt that makes this world possible?

Pretty much everything that we use at Google is open source for our forensic analysis. On the acquisition side, we have a library called libcloudforensics or Cloud Forensics Utilities. Basically, that is a library that wraps around the cloud APIs and lets you easily do things like acquire disk images and Attach them to analysis VMs and stuff like that. We have GRR, as I already mentioned, GRR Rapid Response, which is our endpoint agent that you can use to acquire artifacts live off the system.

Which is also open source, right?

Everything's open source that I'm mentioning, yeah.

Is there like magical reason that open source here is better because some legal scrutiny, or is it just coincidence? Or because we love open source, because we do.

I mean, Google loves open source, but open source also allows the community to get involved and, in my opinion, makes for a better toolset. Different people bring different scenarios to the table and different experiences, and it just enriches the tool with contributions. On the processing side, so once we acquire the data, we have Tobinia, which is our distributed processing engine.

Uh, what's it called? Sorry, what's the name?

Tobinia?

It's got a cute little, like, engine logo, right?

Yeah, it's named after the submarine, I believe. Tobinia is a submarine.

Huh.

Yeah, that's right. Yes. It basically takes the processing task and distributes it across a number of different workers to run things in parallel, makes things faster.

Mm-hmm.

Behind that, you've got Plaso, which is our timelining tool, which basically takes all the chronological data from a file system and puts it into a big super timeline, they call it. Once the processing is done, all this data is then sent to our analysis tool, which is Timesketch. It's a collaborative analysis platform that lets you look through the timeline. It indexes the timeline and makes it searchable. You can tag things and add comments so that other analysts can see where you're up to and continue your analysis. And then there's DF Timewolf, which is our orchestration tool, and that kind of sits around everything. Basically, you can create recipes there so that with one command you can have DF Timewolf go and collect the data, send it to Binia for processing, and then take that output and send it to Timesketch for indexing. And then at the end, it will just give you a link to the timeline and an analysis VM with the disk attached if you want to do some manual forensics. So it makes the analyst's job kind of easy at the beginning. And we have OSDFI infrastructure, is a project where we've created Helm charts to deploy all of these things with ease. So it sets up the whole environment for you, which is quite good. Hmm.

So there's quite a bit of tooling there, but it's a distributed set that you kind of glue together.

Yeah. DF-Tamewolf kind of is the glue that holds everything together.

Nice.

But the skills that are required to do investigations using these tools, I feel like I still see sort of a Venn diagram of forensics, of somebody who's been living in EnCase for 20 years and somebody who knows cloud. And there is some kind of a Venn diagram of classic forensic skills, cloud skills. And sorry for forcing my framework on you. Yeah, I'm a former analyst and all that. But if I think of a Venn diagram, how does it look like? Is it like forensic skills from on-prem mostly survive in the cloud, or is it only a small percentage survive and you need to like learned cloud really well. My question ended up being kind of fuzzy. Is it mostly cloud or mostly forensics? Let me ask you like this.

I would say it's mostly forensics. It's just a little bit of a difference in how to get the data in cloud. But I mean, it's still largely log analysis, which any forensics analyst should be quite capable of doing. And then once you acquire the data, it's just your standard forensics, mostly Linux forensics in the cloud. But yeah, it's There's a little bit of complexity because it's cloud, but any forensic analyst, I think, would be able to quickly pick that up if they spend the time.

That makes sense. So let me ask you a scary question, then. So when I was an analyst, the advice to clients in many cases was that if you face something really tough, you call for help. And of course, I guess at the time, calling for external help means calling Mandiant in some cases. Well, we can't call Mandiant for external help because, like—

Not anymore.

It's not external anymore. Do we ever call for help? And if yes, who do we call? And I think I'm a little scared asking this because it's almost like if you see a volcanologist running, you have to run behind him because volcano is going to explode. So who do we call for help and when do we call for help?

Yeah, who are you going to call, as Ray Parker Jr. would say?

Who are we going to call? Like, if people are going to call us at MainDiet, then who do we call for help?

Who's the adult in the room?

That's a good question, really. We haven't— since I've been here, we haven't had a situation where we've needed to call for help in terms of forensics. My team specifically are the incident management and digital forensics team within Google. So we are the ones that would manage the large-scale incidents and things like that.

So you're the Ghostbusters.

We are basically the ones that people call if other teams within Google have an incident. Yeah, I guess that's who we call.

You call yourselves.

We, we call ourselves, but if the incident is actually big enough, Google has a team called Tech IRT. They're very skilled incident responders. So if we're sort of struggling to both manage the incident and conduct the investigation, we can call those guys in to help us cover the incident management side of things while we get a handle on the investigation.

Hmm.

And then they'll hand it back over to us. So we do have kind of an escalation above us, but it has to be a really large incident to get to that stage.

So it's good to know that it never happened. I think that I'm kind of like, feel really good that you do have a phone to pick up if something really blows up, but you never really picked it up.

That level of escalation to Tech IoT has happened a couple of times, but yeah, it's more just to get a handle on the incident management side of things and let us focus on the investigation. And then once we get a handle on the investigation, we can take over that incident management side as well.

So I want to maybe shift gears a little bit and talk about preparing for this. Surely you're not constantly in incidents, so you must spend some time preparing. What's your advice for, say, a security director at a medium to large corp to help their team be better prepared? What should they do to get ready?

To go back to your point on working out who to call for help, I would say figuring that out and having a response plan that very clearly sets that out is a good step. Because, I mean, depending on things like the scope of an incident, how big your response team is, that kind of thing, it pays to have escalation plans in place and documented so that you know what to expect. Thinking of things like SLAs and response times for people that you will call and documenting a set of criteria that basically tells you, okay, it's time to call for help. Also consider doing like proactive compromise assessments. Because a lot of the time, as a consultant, we would work with a lot of companies who had threat actors sort of sitting on the network for a number of years undetected. And it wasn't until the customer kind of thought to look for them, they realized they were there.

Here, we definitely call Mandiant. Well, people, the recommendation would be to call Mandiant for sure, right?

Yeah, Mandiant, or, you know, there's a number of consultancies out there. But I mean, Google Cloud just acquired Mandiant. That's definitely an option.

It's funny, my mother for a long time worked for a team at AT&T that did network disaster recovery. And I think there, whether it was official or not, she definitely had a mousepad that used to say, preparing for the inevitable. And I feel like that's kind of where people need to put their heads at here. You will have to do incident response and you need to get ready for it.

I have a more specific sub-question to this question. So imagine that it's advice for somebody who was mostly raised in the data center land, pre-cloud era, and they sort of think they're prepared for incidents. They know a thing or two about forensics, maybe have a tool license lying around. So what are the top 3 things to prepare for cloud forensics, specifically for a company that isn't very cloud savvy? Like, what would you learn to be able to apply your forensic skills in the cloud?

That's a great question, Anton.

Oh, thank you.

I mean, I kind of mentioned it. A little bit before, but taking the time to learn about the cloud audit logs and what they mean and how they can help. Some of them may not be enabled by default, so working out what you need to enable— there's going to be a business decision there, obviously, because the cost of storing logs could be greater if you enable more things. But yeah, knowing what they mean and how to read them, how to use them in an investigation, so preparing that in advance is a great idea. Brushing off your Linux forensic skills, definitely, because I know I've worked with a lot of incident response teams who have a great focus on Windows forensics because that's typically what would have been used in on-prem environments. But in the cloud, there's a lot of Linux, so knowing those skills is definitely going to help there.

I once met a newly hired SOC director at a large financial and This newly hired person was for the cloud team and they said, oh yeah, I've got 20 years of Windows on-prem incident response experience. Could you tell me what IAM is?

Okay.

So I agree that the cloud stuff is important, especially for people coming from the Windows side.

Yeah, and then the other thing I would mention is like scaling systems like Kubernetes, knowing how they work and how to investigate within those environments. That's where the complexity comes from, yeah.

So, to start closing this, I feel like I want to abandon nuance and ask you like a painfully silly question. Like, what is ultimately, what is easier? What is harder? Classic data center/PC/laptop forensics in the pre-cloud age or cloud forensics? If you're woken up at 3 AM, alternative 3 AM test, and you have to do forensics on something, would you choose cloud or would you choose the classics?

I mean, maybe an unsatisfying answer.

She's gone back to sleep. Fine, whatever. I get it.

That's not an option.

I might have an unsatisfying answer, but I would say that cloud forensics is kind of easier and harder in ways. Easier because like we already mentioned, evidence acquisition can be much faster and simpler. Got API calls that basically will let you take a disk image and stuff like that. But there's also a lot of useful information in cloud logging. So you sometimes don't even need to forensics if the logs are already available to you there. But then harder because of environments like Kubernetes where instances or containers might be ephemeral. So they may like spin up for a workload and get compromised, but by the time you're ready to investigate, the instance is gone or the container's gone and you lose the evidence that was there. So then kind of log analysis is all you have. Mitigation is also a lot more of a challenge with environments like Kubernetes because You might have a compromised node in a cluster and you isolate that node or you shut it down, but then Kubernetes realizes that your workload now doesn't have the availability that it's supposed to have, and then helpfully spins up another node for you that has the same vulnerability of, of the one you just shut down. So dealing with those kinds of things adds a little bit of complexity too.

Okay. I hate to do it, but I have to ask our actual closing questions now. One, do you have recommended reading for our audience? And it could be anything from Mary Oliver to not Anton's blog. And one tip to help people do better at cloud forensics.

For reading, I would actually suggest Google has an SRE workbook and it's obviously focused around site reliability, but chapter 9 of that workbook is on incident response. It's at sre.google in the resources section. You can find the SRE workbook. That chapter 9 is basically a high-level overview of how we do incident response at scale at Google. And it's a really good framework basically for if you do have a large incident and you're trying to get it under control, it's a known framework that you can train your team on so that when an incident does occur, you have some familiar structure. It just helps you bootstrap the incident really quickly. That's something that we find really helpful here at Google. As for a tip, we've already kind of mentioned it a few times, but preparation is key. Don't be trying to figure things out after an incident does occur. Basically, take the time to learn about your Cloud Audit logs, what they mean, how they can help you in an incident. In your plan, incorporate blameless postmortems as well. That's something that we have built into our culture here at Google.

That is a great part of our culture.

Yeah, basically, at the end of an incident, make sure that You write up a postmortem, look at all the things that went well, things that didn't go well, where we got lucky, and then take the key learnings out of that and make your process better for the next incident. And then, yeah, finally, I guess, brush off those Linux forensic skills.

I love that. Well, Jason, thank you so much for joining us. This was delightful.

No worries. Thanks very much for having me. It's been a pleasure.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, Or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikian and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.