EP103 Security Incident Response and Public Cloud - Exploring with Mandiant

Security.

Hi there. Welcome to another season of the Cloud Security Podcast by Google. Thank you for joining us today. Your hosts here opening season 3 are the hosts from seasons 2 and 1, myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, A now quite tamed and reformed analyst, and now senior staff in Google Cloud's Office of the CISO. You know, listeners, when we started the show, Anton was fresh off the analyst beat, and I've noticed over the last 2 seasons, he's gotten much nicer to our guests. I think you should all encourage him on Twitter and LinkedIn to, to go harder on them. You can find and subscribe to the show wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you like our content and want it delivered to you piping hot every Monday, please hit that subscribe button. You can follow the show, tease Anton, and argue with me, twitter.com/cloudsecpodcast. Anton, we're kicking off the season with something that I hope most of our guests never have to deal with, which is cloud incident response.

Yes, we are kind of coming back to this topic because we did cover it in one episode, but the advantage here is that we are talking to somebody who kind of lived through many probably painful, maybe some of them were painful cloud IR experiences. And it really means that he has seen scary things, and he has seen things done really well. At least that's my suspicion, right? So, we can hear some failure stories, and we can hear some success stories around incidents in the cloud.

Yeah, he's heard things go bump in the night that nobody wants to hear go bump in the night. So, perhaps with that ominous introduction, listeners, Let's welcome the first guest of season 3. Today we are joined by Nader Zaveri, Senior Manager of Incident Response and Remediation at Mandiant, now part of Google Cloud. Nader, this is going to sound like a bit of a long-winded way of getting to the question, but one of my all-time favorite blog posts— remember when people had blogs instead of Twitter threads? All-time favorite blog post from this guy, Ken Norton, who wrote an article called How to Work with Engineering. And it's just a whole thing of anti-advice for PMs, you know, always be late to meetings, always tell engineers how to fix the bug, that sort of thing. In the spirit of counterexamples being powerful advice, do you have a good story of cloud incident response going wrong? I think that's always illustrative of how not to do something.

Yeah, definitely. Thanks for having me on, Tim. Part of being in Mandiant as a foremost kind of incident response firm, we have over about 1,000 incident responses that we do respond to a year. And a lot of our people and personnel that we have are great incident responders when it comes to on-prem incident response, things like host-based forensics, things like network type of analysis. I think one of the things I would say is early on during an organization's cloud journey, and also when we started realizing a lot more cloud incidents were happening, just because you are great at doing host-based forensics and network-based forensics does not exactly one-to-one equal that you're going to be a great cloud-based forensics person, because there's a lot of things that are happening in the cloud, especially at the management plane level, that sometimes does not get into some of the logging that you'll have at the host level. Another thing was just how much logging is needed within a, let's say, cloud-based incident response. When you're doing a traditional IR, in this instance, we'll call it a traditional IR, which is more of an on-prem Active Directory, the attacker's able to abuse Active Directory and start having a C2 channel going out to their command-and-control channel. There's a lot of things that we glean from host-based forensics, from, let's say, doing the live response at the host level, that, you know, an organization may not have the logging turned on, but a lot of things are maintained at the host that an attacker was able to do without the organization having to turn on a specific set of logs. Yes, having logging and event logging and these types of logging does help out our incident response, but a lot of times—

But there are also flat-out bad logs. I mean, occasionally I see logs from some cloud provider that just missing something really critical. I don't know, somebody connects to a machine, but there's no username, or some comical stuff like this. I mean, it does happen.

Yeah, exactly. And I think another thing that we've seen is— and what we've been doing throughout the years is, as we're doing cloud-based IRs, if we're starting to see certain logging not— even if they're enabled, there are certain key attributes that aren't being logged, we go back to the major cloud providers, the AWS, the Azure, the Google Clouds, and try to say, hey, the attributes that are not being logged, this is something that, if you have this information in your metadata, something like this would help out an organization better understand it. And we're helping out also at the cloud-based level. And another thing I would say, a counterstory or a counterexample of, is make sure your IR team has access to the cloud environment. Usually—

You know, that helps, right, if you're doing cloud IR?

It'll help a lot. And it's too many horror stories Where we get into an IR, it is, let's say, an AWS or GCP-related IR. We're requesting access to the specific cloud provider, and the IR team doesn't have access, doesn't know how to provision access. And then now we're talking to the cloud operations people who don't have anything, no idea what an incident response is. They're purely cloud operations, which is fine. That's what they do.

Mm-hmm.

They need to give us access, and we need to start asking specific questions about the incident, what they know. And now it's a— we're teaching them incident response and some of the terminologies that we're using that a typical incident response team would know. So I think one of the big things is, and this is really the bane of all security people's existence, is when a lot of organizations are making their cloud journey, the IR team, or really just the security team, is the last to the party. Unfortunately, making that journey because DevOps and the agileness of the cloud allows a lot of organizations to push into the cloud more agile with a lot of ability to create applications fairly quickly and spin up instances. And security is usually the last to the party. And that's what we've seen.

But this is an IR episode, not a cloud migration episode. So let's, let's stick with the IR side of things. Anton, you had the next question, right? Yeah.

Yeah. I was just being slightly depressed that the first challenge for for cloud IR is, IR team doesn't know where cloud is and can't get there. Like, this just strikes me as kind of bizarre, but at the same time, I totally get that it's actually common. So, it's one of those kind of common sense is the rarest of all senses, right? It's like common sense, you should get there to do IR, but in reality, you can't get there. So, on this depressing note, are there examples of people doing it right? What could that team have done right? So, are there particular skills they should have developed? Are there other things?

Yeah, I think one of the things is ensuring that you are not only enabling the logs adequately, but also pushing those logs into— so throughout the years, a lot of organizations have really made a robust transition into enabling on-prem logging and consolidating and pushing that and forwarding into a SIEM or SOAR solution. A lot of times when you're seeing it from the cloud instance, those logs are living in the cloud. Or even if they're forwarding those logs into that SIEM or SOAR solution, those cloud logs aren't tokenized. The attributes are not correlated with the on-prem attributes. So it's just random raw logs being dumped into a SIEM/SOAR solution, which from an incident response standpoint, if they're not tokenized, if they're not able to be clearly, easily queried, just makes an incident response response team harder. So one of what we've seen a lot of organizations who are very mature in the cloud and have also put in the time and effort and personnel training into not only training up your IR people to also respond in the cloud, but also creating those types of logging and metrics that can be gleaned during an IR if, if there is an IR that is in your organization.

So how do we want to do that training? What does that look like? Is this tabletops? Is this red teams? What's the right way to train your IR people to get good at cloud?

So one of the things I always say is your IR people need to at least understand the management plane of the cloud environments, understand the major services and resources that are happening. You know, we're not expecting them to be cloud operations people, but understand what is happening in terms of the management plane, in terms of the logging that can be enabled and the things that can be gleaned. But really, You know, having tabletop exercises is a great idea. So having cloud-based tabletop exercises where you're focusing on a common cloud-based attack, like an exposed secret or a service account exposed to the public that now a threat actor is able to utilize and then get into your environment and then wreak havoc. Or public storage bucket. Inadvertent public storage is something that we see a lot, or even advertent public bucket being exposed and compromised. So having these types of tabletops that are really catered to specific And, well, you know, there's an executive-level tabletop, and then there's a technical where you get into— you show them exact logging from various different log providers and cloud providers like AWS and CloudTrail and Stackdriver, things like that, types of logging, and then walk through what would you do in this situation, where would you look at for logs, and then really be able to test your incident response team on how they would respond to a situation. to respond, and what is a correct way to not only respond to the incident, but also remediate and mitigate these types of situations.

To kind of switch to something possibly a more fun topic, are there maybe 2 or 3 specific instincts that maybe even good incident responders have that would get them in a completely wrong spot in the cloud? Like any particular on-prem instincts or, you know, intuitions that will lead responders astray?

I wouldn't say they will be led astray. I think one thing with incident responders, I would say always keep with your instincts. Those instincts stay the same. You will just need to— because the threat actor will use the same methodology when they're going through their attack lifecycle. The methodology, if you want to focus on the kill chain or the MITRE ATT&CK methodology of a specific threat actor, is going to be usually the same. The method— what exactly is happening changes a little bit. One thing I would say is the instincts stay the same, but understanding what you can glean from the management plane of the cloud environment is where a lot of organizations need to really up their game with some of their incident responders for the cloud.

Okay, so let's get into the tech of all this. Anton and I love to say that you can, you know, leave your firewalls out in the great bucket in the sky and, you know, your cloud environment is good with the native tooling. When it comes to what we know from on-prem, what we know in the cloud, what are the on-prem things that can just stay there and not come with us to the cloud?

So, an example of EDR comes up only every other time when people show up and half of them say, I'm going to the cloud and I'm taking my EDR with you. And the other half says, whoa, what do you mean EDR? We use microservices. There's no E. And they all look like they're obviously right, but their views are ultimately opposite. So, that's an example where The tool— we don't know whether the tool should journey with us. You know, we know SIEM should, but we don't know that EDR should. Like, that's an example.

No, that's a great example. So one of the things— so with EDR, if you're using that as an example, I think even during our incident response services during a cloud-based attack, if we see the attacker traverse, let's say, a virtual machine or an EC2 instance or stuff like that, we will be utilizing EDR. So having an EDR solution, especially those that are cloud-capable, rather than just only on-prem level attacks that are happening at the normal on-prem level. Those are definitely something that I would say can come as long as that organization or that vendor that you're using has input the time to create their solution to be cloud native in a way, to understand what's happening in the cloud. You know, as you mentioned, you know, the cloud is evolving.

Yeah.

Firewalls, network access controls, all that can be implemented natively in the cloud. So a lot of these major vendors are now creating kind of partnerships and creating Zero Trust partnerships so then you can implement Zero Trust throughout the cloud environment. But really, you could do a lot with just the native tools. And I think that's one of the big things with what we've seen is there's a lot of native capability that happens in a cloud environment. that even at a cloud operations level, the response team doesn't know. And, you know, one of the things I think what we see a lot happen or used to happen a lot during the early days of our incident response services is we used to do a lot of deep packet inspection, a lot of network logging inspection. That isn't as much needed, right? So really NetFlow data is really sufficient when you're doing kind of VPC flow logs within an incident response network.

So you're arguing against full packet capture in favor of flows? Like, you're—

I wouldn't say arguing against. I would say there's still some times where we maybe need some packet mirroring capabilities, but very rare. It's not as needed because the amount of data that gets gleaned from VPC flow logs and whatever we can glean from the management plane, from other types of logs and access logs and admin-type logs within, let's say, Google Cloud or AWS, we'll be able to tell a good story and we'll be able to help the organization understand the kind of nature of the attack, as opposed to having to do those deep packet inspections on 1 or 2 packets that are happening in the cloud.

So you sort of went through some tool categories. What about some processes and practices that you perhaps won't take with you? So I don't know, I'm thinking about asset discovery being completely different now because instead of, you know, snooping around closets or looking for wireless devices, you go scan APIs or whatever. But are there particular security or maybe IR processes that you don't take with you when you go to cloud?

Yeah. So I think asset management is just changed totally in the cloud, right?

It's just different. Yeah.

Even account management, finding out the source of the account, what that account can do, and what does it have access to, is not something we have to ask people anymore, right? We have that type of data. within the cloud platform, within the management plane, we have logs of that account and the permissions, the API permissions that has happened with that account to understand if that account has the capability to perform such actions. So yeah, definitely I would say account management is also another thing that has kind of drastically changed. And really, the old adage is identity is the new perimeter to the cloud and how much identity is so especially with the other types of cloud providers, it's one of the main building blocks of the way the cloud and securing the cloud is really made up of.

So this all makes a ton of sense. When I was a Boy Scout, at least until I realized my queer self wasn't super welcome at the time— they've gotten better— I loved their motto of always prepared. How should teams prepare for IR in the cloud?

First things first, it may sound like I am self-promoting, but If your organization does not have an incident response retainer, you definitely want to get an incident response retainer with a reputable IR firm. It doesn't always have to be Mandiant. We'd love to do with Mandiant, but with other reputable incident response firms. Also, having cyber insurance is very key. It helps especially with ransomware situations, which, you know, there is capability ransomware in the cloud. It's something that is not as common, not as normal. It's mostly on-prem, but these are all things that I think an organization should do from a organizational standpoint, but also, I think we talked about it a little bit earlier about tabletop exercises. One of the best ways is— it's really the cheapest way— is to test your IR teams with tabletop exercises. There's also simulated things like a security validation, where certain tools out there around various vendors that can simulate a specific threat actor group. So once you understand your threat landscape and you understand the Threat actors that are targeting your specific industry. A lot of those simulation-type of exercises where they can simulate a specific threat group and their TTPs related to that, that can help an organization understand the type of logging they need to embed, the type of preventative measures that did not stop it that they may need to enable. Those are all kind of things that I feel, from an organizational standpoint, You have to have. And not only that, so we talked about logging, and we talked about consolidation of your logs, but ensuring those logs are correlated with your on-prem data. So when a threat actor is logging into your cloud environment using a— let's just say using Bob User as a username, is that correlated to something that's happening on-prem that we can see that something like Bob User was then pivoting from the cloud down on-prem or vice versa from on-prem up to the cloud.

So, given this conversation, I wanted to switch gears a tiny bit. I guess maybe not to my favorite, favorite subject, but one of them, namely logs. So, do you have any absolute favorite log sources for this craft, for IR in the cloud? So, if somebody wakes you up at 3:00 AM and says, I only have time to enable 3 log sources, what should I do? And you're still asleep. So, what would you tell them?

Listeners, don't be fooled. Logs are definitely Anton's favorite topic. There's nothing that compares to logs for him.

Okay, fine. They are.

Yeah. So I would say from a cloud standpoint, VPC flow logs, number one.

Wait, even before Cloud Audit admin activity logs?

Not before that, but I would say a lot of that is—

Ah, okay. So number zero.

Let's have a log fight.

No, that's a great point. So I'm saying VPC flow logs because a lot of organizations already have the Cloud Audit. audit admin logs enabled by default a lot of times.

Ah, which are enabled by default in Google Cloud Listeners.

Correct. But the VPC flow logs is something that an organization has to enable. And in a lot of organizations, and it can get fairly pricey. And we understand that it's, especially if it has a high traffic, high volume, it gets— so being able to understand which VPCs houses your crown jewels for your organization, those production-level type of data, And those critical data and enabling just those flow logs at least, right? So being able to prioritize that as much as possible. Because storage buckets and S3 buckets are very, very popular for threat actors, having deep logging within the storage buckets or S3 object level or access logs is another big thing that we're seeing that if you were to turn this on, a lot can happen. I mean, I can tell you, Just from an IR that happened recently where, within AWS, we saw the threat actor access a specific storage bucket, S3 bucket, and we know that they accessed it, but they didn't have the object-level logging to see all the specific API calls, how much data the threat actor was able to actually access. And because of that, with Council, you know, when you're doing that victim notification, when you're doing the notification to everyone, from, let's say, a bucket that houses 100,000 records, the threat actor really was maybe accessing 30,000 or 300 records. But because we didn't have that deep-level logging capabilities of figuring out which specific objects the attacker accessed, you have to go with they accessed the bucket and then assume that they've accessed everything because you don't have that data, the evidence to prove otherwise.

Well, that makes sense. I can see the challenge of proving a negative if you don't have the logs.

Exactly right. It's impossible to prove a negative. Another, I would say, logging that's not commonly thought of is really the load balancer logging. That helps us out a lot, especially during IRs when let's say they didn't turn on VPC flow logging. We can then go into some of the load balancer logging they have, and we've had some great success with capturing the threat actors from there as well.

That makes a ton of sense. Well, I've really enjoyed this, and it's always fun to disagree about logging, but I'm afraid I have to ask our traditional closing questions. One, do you have recommended reading for people interested in learning more about IR in the cloud? And 2, do you have a weird tip? And, you know, no pressure, but this is the first weird tip of the year.

So I would say reading recommendations. Well, there's so much to do in terms of reading, and One of the things is understanding the logging that are enabled across the cloud platforms. So, for example, Google Cloud has very deep, intensive understanding of what type of logging is enabled by default versus what is going to be— you have to enable by the organization. AWS has a great paper on just all the different logging, but also they have incident response paper on how to approach incident response. Another thing would be For, of course, Azure, they have a lot— Microsoft has a lot of documentation on specific logging. So understanding the types of logging and the type of data you're supposed to glean from those logs. I'm a big follower of a lot of great people on Medium. So once you have a chance to get Medium and, you know, associated with whatever cloud platform you are, you really get a lot of great information from various authors and you'll be able to start gleaning yourself which author you like in terms of who is the person you need to follow for AWS security and AWS logging and GCP logging, GCP security. And really, I would say from in general, a lot of their cloud providers have built out great white papers and being able to understand that, what white papers to really look for within each of the cloud providers.

Awesome. Well, Nader, thank you so much for joining us today. This has been a real pleasure.

Thank you for having me. Thank you.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @anton_chuvakin. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.