EP98 How to Cloud IR or Why Attackers Become Cloud Native Faster?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, former analyst and Senior Technical Security Staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to your device every Monday afternoon Pacific time, please hit that subscribe button. I'm also going to take this opportunity to please thank you to rate our show highly on your podcasting app of choice if you like our content. If you don't like our content, please don't rate it. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we're talking about— incident response today, and I did not know until we recorded this episode that there was an actual 3-part definition of what counts as an incident. That was eye-opening.

Yes. And it comes from a broader field of incidents. I think the disaster recovery, even outside of IT. So I think that that source of wisdom wasn't security, hacking, malicious, whatever, or even IT incidents. It's came from outside where you think about disasters in the physical world, which to me was kind of fascinating because I was about to argue with a guest about one minor point and then I realized, well, actually this definition kind of removes this reason to argue.

Yeah. The topic is near and dear to my heart, listeners. Fun fact, my mother for most of my childhood was part of AT&T's disaster response organization. So I spent a childhood climbing around DNR exercise locations. So it's very dear to me. Today's episode is a ton of fun. We have returning guest John Stone and a new guest. With that, let's turn it over to both of them.

Welcome to the show. Today, we have Matt Linton, a chaos specialist at Google, and John Stone, chaos coordinator at Office of the CISO, Google Cloud. So all that much chaos, what is the topic? Obviously, the topic is incident response in the cloud, as things happen. So back in 2014, when I first touched on this topic, to me, the number one challenge was getting the data to investigate, as some of the cloud providers 8 years ago had few logs accessible to clients. So, what are the top 2022 cloud IR challenges? Let's start from the problem space. Matt, John, whoever wants to answer first.

Oh, that's a tough question. I mean, I think the challenge varies per company or per individual even, right? are some places who are just beginning to adopt cloud, and, you know, their challenges are all in either responding to ad hoc builds that have been put together or, you know, formalizing things that were just experiments a week ago.

Mm-hmm.

And then there's other institutions that are really large, and they're bound by regulations, and they have mature processes, and their challenges are about how to actually meet the same kinds of processes they have in place now, but in the cloud, right? With different resources and different risk profiles. So it's hard to even say anybody has kind of a one-size-fits-all risk profile there.

But to me, it sounds a little bit like everything is top challenge.

Well, I think there's a varying maturity model there. So maybe John can lay that out for us.

Yeah, I'll add these 2 buckets. I'll use the owl analogy, the whole OOT thing. And for the folks that don't know what OOT is, it's organization, operations, and then technology. So, and I think it falls into the first 2 categories, right? So, the organization, depending on, like, what Matt was saying, like, either is very new to it or doesn't have that. And a lot of times, they don't have the operational capability. So, like your 2014 example, Anton, I don't think it's an issue of having the logs anymore. In fact, there's probably too many logs right now for them to look at. But it's the, do they organizationally have the skills in their organization to look at cloud? A lot of times they don't. And then do they have operational capabilities? But everybody wants to run to the tooling first, which is the T of the OODA thing. And that's normally the one way it goes wrong from my perspective. Like, if they just approached it from the 2 other things, like answering the question of, can we even do it before Here's the thing we should do it with. I think that would solve a lot of the challenges. I think it falls in those first 2 buckets.

It sounds like very early-stage problems of a domain. I'm trying to put my analyst hat briefly on and think, this sounds like problem stamp what you would describe for a domain that's really immature.

Well, define immaturity because, I mean, cloud has been around for more than 10 years already. So, it's not a new thing. But some organizations have been slower in moving to cloud. Now, if you look at a digital native organization, well, they were born in the cloud. It's not a new thing for them.

Mm-hmm.

They have a different way of approaching this whole problem. In fact, the only incident response they know is in the cloud. But if you look at the larger scale enterprises, that's a brand new thing for them. That's a new thing where they're pushing to, and especially regulated markets are even slower to that uptake. Around that. So it's a sliding scale. I wouldn't say it's like a starting thing, but to Matt's point, different customers are at different points in their journey.

So part of the problem here is just knowing yourself and knowing where your organization is in that journey. When we asked that question, we kind of did something unfair. We didn't start out by saying what a cloud incident even is. So Matt, when we think about security in the cloud and incident response in the cloud, What is an incident in the cloud? Because things like exposed storage buckets or crypto mining attacks on an exposed project, that happens pretty often. Do those still count as incidents? What's the line?

Well, you've got me thinking very philosophically about incidents. And I should say, you know, I majored in disaster management and I like to think philosophically about incidents. So if I start you from the ground up, you know, we define an incident philosophically as needing 3 necessary components. It's something unexpected, you didn't expect it to happen, it has a negative consequence, and you don't already know how to fix it quickly. So an incident, you know, with a capital I is something that you need coordination and a team of people or, you know, processes and procedures to fix. But, you know, of course a lot of people see an exposed cloud bucket and what's in the bucket depends on how bad that impact is.

Mm-hmm.

It could be an event, Because someone built a web page and they're literally using a public bucket to store the files as sort of a starter CDN, or it could be entire employment records from a government agency who configured the bucket wrong and there's a ton of PII in there. I would definitely consider the latter an incident. The former obviously is intended. So I think you need those 3 necessary components, but beyond that, it's what's valuable to your organization and how much maturity you have in place for how you're going to quickly detect it, quickly revert it, and know what needs to be done after that point.

So just to make sure we got that, an incident is unexpected, it has negative consequences, and there's not a clear fix immediately available.

Yeah, that's how we define the kind of incident where you would get into incident response plans and kind of incident management.

Yeah, I was actually about to prod at this, but actually the last argument, the third of the pillars, kind of disarms my question because, of course, open storage bucket here is pretty clear how to handle it. But say, in the past, before cloud, I always objected to people treating critical vulnerabilities as incidents because I felt like, well, it's a potential incident if somebody compromises you through that vuln, but a vuln itself isn't an incident. But in a broad scheme of things, it is kind of an incident because you are rushing to fix it and there may not be an obvious fix for it. So, it sounds like you're going broad and that's okay. In the cloud, we tend to go broad, right?

Yeah.

That's a good case study example to kind of go through. And ideally, you're living in a world where vulnerabilities aren't incidents because they're not necessarily causing you any harm. So let's take a critical vulnerability in something as a good example of that, right? It's unexpected, that's check. If it's protecting data, there's a potential negative consequence, but is there really a negative consequence if you have good compensating controls, right?

So even if you have good luck— If you have good luck. You have no—

Right.

Parts to exploit, hard to find.

Part of this is preparation, right? Like, we make a joke sometimes, there's 2 kinds of people, those who sandboxed Ghostscript and those who should have sandboxed Ghostscript. And that's kind of a nod to, if you have layers of compensating controls, vulnerabilities aren't incidents, they're events. If you have no compensating controls, vulnerabilities can be incidents because you don't know what's happened and you don't have anything to depend on. Another kind of easy-to-remember thing that I always tell people is the prepper's motto, 2 is one and one is none. That applies to layers of defense as well in kind of the security model.

Anton, don't you know the Thomas Jefferson quote about luck? The harder I work, the more I have of it.

Yes, yes. And of course, in my former motherland in Russia, it's attributed to some ancient Russian author, obviously. But yes, I do know the quote.

Oh, perfect. Does it rhyme in Russian?

It does not.

Well, Tim, I just wanna add in just to Anton's point. I just wanna bring a completely left-field example as well to also illustrate that a toaster has a vulnerability and it's called water, right? But unless I get in the bath with my toaster plugged in, it's not an issue.

That's your vulnerability if you go in a bath.

Yes, we are. That's also my vulnerability. So then we're commingling 2 vulnerabilities. But if my toaster's plugged out and I get in the bath, Yeah, the vulnerability's still there, but there's no impact. So, in incidents also, the scale of the impact, like Matt mentioned, you know, like, that's a critical factor to those kind of things as well. And one of the things that we don't forget about cloud is— we shouldn't forget— is there's also automations and things to do things faster. So, imagine you have the scenario, this theoretical scenario of uploading a private key into a public bucket. But you have the ability to respond in, like, let's say, 30 minutes or less, and it's gone. More than likely, the impact doesn't exist because you took it down before it became an issue. So that's also a consideration with that.

Okay. So that's a really interesting angle here, the automation and automated response. Should we do IR differently in the cloud? Like, Matt's talked about preparation and no clear fixes available. Do the plans for finding fixes during an incident need to be different in the cloud? You know, what changes organizationally or operationally or even tooling-wise? Because you mentioned automation, that sounds like tooling to me.

What I would say there is, I think the single biggest problem I see in organizations who run into not being able to respond in their cloud environments is lack of preparation. And that is the same mistake we see in corp environments, right? People didn't realize they needed to do IR until they realized they needed to do it, and now they have no endpoint agents and they have no logging and they don't know where to find things. I think in cloud, what you see a lot is people building a cloud environment and not having response plans, either because they didn't know they needed them or they were genuinely small enough they didn't need them yet. And then they need logs, but they never configured the logs in the provider side of the house, right?

Mm-hmm.

They need access to all of their hosts at once, but they never configured management tools or endpoint agents. The number one advice I would give people is, yes, you should definitely have a plan, but it all falls under the same kind of preparation for the unexpected happening that you begin to put in place when you begin to build response plans.

And I take it even a step further. I've unfortunately seen some of this from the front lines. There's a— I think it's a Mike Tyson quote, and if I'm wrong, he'll probably punch me about it, but He says something like, everybody has a plan until they get punched in the face. And the very funny part of this, a lot of times when I've seen incident response teams in cloud act on an incident, the very first time they see a console or a gcloud command or whichever thing is in the incident.

Oh no. This is kind of depressing though. It's a depressing thought of the day.

But that's to Matt's point. That's that preparation. So a lot of times it's not, changing initially your plan of how you would act in an incident, but it's how do you adapt to that new environment, to new tools that you don't know, things that aren't documented. I've seen scenarios where people have tried to get disk copies out of an environment, and they have controls in that stop you from copying anything out of the environment, and they spend 2, 3 hours figuring this out. Well, if they had prepared in the firsthand and they documented, they knew when the incident rolled in. This is what it looks like. Here's what we need to do, and those kind of things. So, it's not necessarily doing it differently, but it's actually practicing those things.

Yeah. It's different to drive in the sand, and if you haven't practiced driving in the sand, you're going to have a bad time.

Even though you can drive.

Yeah.

Right. And in my Gartner days, I love to kind of highlight the fact that it's plan versus planning. Like, you have a plan, I get it, I'm happy for you, but are you planning? And this felt like many people have a plan but don't do any planning.

Yeah, and I can't stress enough how important the pre-planning is, especially turning on logs in cloud environments, right? There are a lot of cloud providers out there, and many or most, I would say, provide relatively okay logging, if not downright good logging, if you turn it on. And I think my peers at a lot of other companies, you know, have discussed this with me at conferences. The headache is a customer comes in and they say, we have an incident and we don't know what's going on. Can you give us your backend logs because we forgot to turn logging on in our environment?

Yeah.

And it's like the logs that are provided to the customer in the customer console are literally the best logs you could possibly get for investigating an incident on the customer side. Even if your cloud provider has backend logs, they are going to be in a different format meant for quality assurance or meant for uptime management or meant for reliability. And even if you manage to get logs from the backend of that provider, they're not going to be the kind of logs you would want for a security investigation, as opposed to thinking ahead and turning on the logging. But I do have to acknowledge, orgs are in a different position in their lifecycle. Logs, especially if the provider is charging for logging, right, you have to budget for that. You have to budget for how many you want. And we, from large enterprises, we think security incidents are a potential existential risk to the company. But if you're in a startup, everything's an existential risk.

Yeah.

Running out of money next month.

Startups are constantly a week away from nonexistence until they're established. And so I want to recognize that these are decisions that have to get made by leadership, not just things we take for granted.

By the way, I just felt this like empty, hateful feeling in my heart when you said they charge for logs and I'm like, ah, I just felt this blast of hatred in me. Of course, it's okay to charge for log retention, but to me, charging for log enablement, which ultimately leads you to charge for log retention, it kind of, should I say evil? I feel like it's kind of an evil practice. I don't know.

I think it might be a good time to move on to the next question.

Okay, move on to the next question. Otherwise, Anton's going to go off the rails. Perfect. So one thing that comes out a lot, of course, with all the cloud security challenges is, Shared responsibility model, people don't get how to share what. But here we are running IR. And of course, IR here would be a joint exercise with the CSP and possibly also with third parties, like, I don't know, Mandiant, a random name comes to mind. So what's our advice for how to run incident response jointly with a CSP like ourselves and/or possibly including third parties? Just give us a few pointers, like don't write a 50-page document. Well, I'd love one, but give us the top 3 tips here.

Well, I'll start, and Matt can chime in from that perspective. But I think a lot of times, and I'm speaking from a generalization, but things I've seen a lot of times are things that the customer can actually resolve themselves. All the tools and capabilities are there for them, but they're not familiarized with those things. So I'd say that the number one thing is work with your CSP to familiarize yourself with what there is. And how it's enabled and start doing that preplanning, thinking of, like, do I have an environment ready that's for evidence collection and all those kind of things? How would I do those steps in the cloud kind of thing? That's going to get you, like, probably almost all the way there kind of thing from that perspective. And then having your playbook updated, like, how do I contact support? How do I set the right priority in events and things like that? I can't tell you how many times I've seen that, like, kind of fall down. Those are things that is already great stuff to do and work jointly with the CSP in establishing those things from that perspective. I'd give those as 2, like, top tips to start with from that aspect. So familiarize yourself and then update your playbook of how would you contact support, who would you contact, and all those kind of things. That is very vital.

Matt, I'm curious if you have thoughts on that one as well.

Yeah, that's a good answer. I think John reminded me of a separation that we do here in our own internal operations, where we have 2 different kinds of docs. We have a playbook and we have a response plan.

Mm-hmm.

And a playbook is focused on, like, how to get a particular piece of data. You go here, supply this key, run these commands, put the data here in this bucket. A response plan defines organizational interdependencies and kind of the expectations of different teams as they interact together.

Mm-hmm.

And I think one thing that a lot of orgs are missing that you're asking about here is what's the response plan? If you have the CSP, the customer, and Mandiant, who's responsible for which bits? And, you know, in larger incidents, a thing that is almost always a problem is communication and expectation setting. And I think knowing what's expected of you, knowing what you expect of all the other players, and having alignment on that in advance is probably, again, it's preparation, but it's the number one most important thing you can do.

One of the things that that reminds me of on the point of communication, I've seen organizations get in trouble when we happen to reach out to them and don't find the right technical contact. And so understanding as you're getting ready for cloud who your actual technical points of contact are, if you get that phone call in the night from your CSP, that's very important. Yeah.

Seeing security people post on Twitter and say, does anybody know someone at Followed by company name is always— yeah, it's kind of scary because that's like your last resort of getting a technical contact, and it means a lot of other things.

Yeah, but it happens a fair bit though. It's not like super uncommon.

No, but I would also add the flip side is very important, right? Um, and this is one, when you were speaking, this reminded me of this as well, Matt, is a lot of incidents that I've seen in clouds have happened and it shouldn't have happened in the sense that people are starting a cloud, they have a development environment, they may be a large-scale enterprise, and they, you know, starting to build their muscle memory with cloud. And then they set up alerting, they do the right things, but they send it to an unmonitored inbox.

Oh yeah.

That nobody owns or maintains. And then, like, if you do postmortems on some of the events, there's like a couple of days before somebody actually finds a person to notify, you know, and then you see like 2 or 3 days into that event is the first compromise happened. But probably if somebody just monitored the inbox, they would've stopped it immediately there. So it's dual-sided as well, you know, is making sure that if you set up an alert, somebody's actually looking at the alerts as well.

And I would take this as an opportunity to remind all of our GCP customers about the value of essential context. And configuring that correctly for your organization, and perhaps even making sure that you're using Google Groups as the essential contact and ensuring that there's always members of that group that get those to their inbox. So I want to talk about our Threat Horizons reports because, God, that's so fun to say. Threat Horizons report. It's really fun to say. I credit to whoever came up with that name. Can we talk about some of the incidents we've seen lately? You know, what are the bad guys actually up to? John, Matt, either one of you, what are they trying to do?

Well, Tim, they want money.

They want money?

Wow. Surprise.

Shocker. How are they getting that money?

I'm joking only a little bit about they want money, right? A lot of automated attacks these days are focused on gaining access to your compute resources and coin mining on them. So you get the coins and the customer gets the bill. Speaking for the more motivated attackers, right? The kinds of top-tier attackers who want something other than to just coin mine on your back or ransomware you. That's kind of a different set of groups. In terms of goals, you're typically looking at either automated attacks taking advantage of common mistakes, like you publicly posted an access key to GitHub and people are constantly looking for that, and they're going to do ransomware or coin mining or other things that are automatable, and they may never even know they hacked you. As long as they're getting the coins, they're happy. The other types of attackers are typically trying to be cloud-native and obtain persistence.

Mm-hmm.

And again, a lot of that is going to focus around credential management or lack thereof, right? They're going to find a credential, find a service account, leverage it into an admin-level service account, leverage that into creating their own service account they own so that if you revert the key to the service account you thought was compromised, but you didn't notice they created their own, they still have that level of persistence. So kind of top of mind for me are those 2 things, either losing access to your credentials or not maintaining your cloud hosts and then getting worms in there. Or not having the visibility you need to do identity management well and not being able to detect the persistence?

I mean, I don't want to sound like a cliché or make a cliché in this case, but like IAM, cloud IAM, maybe I shouldn't say it gets you every time, but it gets a lot of people every time. Like ultimately, they lose the creds or they mismanage the creds or they over-permission something. And there's a long, long list of IAM Stuff that gets you, right?

Yeah.

I mean, I think the challenge the industry has is the attackers are becoming cloud native very quickly and large enterprises adopting cloud are not moving at that kind of speed. And so there's a power mismatch almost.

This is a great title for an episode, by the way. I need to write it down.

Power mismatch.

No, no. Attackers becoming cloud native faster. That to me is kind of a good reminder. Yeah.

Yeah.

Well, even if you look at the threat intel reports on that one, Anton, you'll see like I think in the first one we, we spoke about crypto mining as being a big thing around that. And then I think it's around the third one where we actually started to speak of how some of the tactics had changed. So they went from straight-up crypto mining to doing like a kind of proof of capacity and proof of space and time kind of things. So they even like changed their tactics based on what was available as It's always that race as the defenders get better and shut down some of those things, they also change and adopt new technologies. So that's like a trend that you also see to that point where they're starting to also become more cloud native.

Well, before we get too far into the doom and gloom of attackers getting smarter and cloud native faster than us, it's time to wrap up with our traditional closing question. So Matt, first, do you have a tip to improve your cyber incident response in cloud? And do you have recommended reading? And then same question for John, but Matt first.

Yeah, at the risk of sounding like a broken record and then also dating myself for the parts of the audience who have never held a record before, it's preparation. You know, I'm a detection and response person, and that means a lot of what I do focuses around finding out what bad things happen and reverting them. And for me, that means, you know, having all of the logs available in advance and having people who are able to look at them and able to find that one service key that doesn't fit in with the others. And then having the planning in place to know how to deal with it and what we're going to do. So I just say preparation and planning is key to almost all of the rest of success in DNR.

Do you have recommended reading? And it's a dollar in the swear jar if you say the SRE book.

I will pay the dollar. Oh wait, no, I'm not going to say the SRE book because it's the SRS book, Building Secure and Reliable Systems. So I get to keep my dollar.

You get the dollar.

Yep.

Chapter 17 in particular, Crisis Management, describes how we do incident response at scale, and chapters 14 through 16 have a lot of the preparation aspects that I keep hammering on here. So SRS, Secure and Reliable Systems.

Well, I think you get the bounty from the jar for having chapters to cite instead of just—

Oh, that's right. I think so.

John?

Yeah, so obviously Matt's taken that one, so I'll put a dollar into the jar for that, and then I'd say you guys take that dollar And then go buy a book that's called 101 Ways to Insert Expletive in the Woods, so you can figure out what that means. And it's basically—

Do what in the woods?

Uh, I think we got it.

Think of the ice cream emoji, Anton, and then you think of that it involves toilet paper. Oh, now that sounds very funny, but the reason I'm saying that is when you gotta go in the woods, You need to have a plan and you need to be prepared. Otherwise, you can get yourself into really, really weird situations. So, that's a lateral thinking book on this kind of thing. If you apply a lot of those principles, then probably you bring that into an incident and then you're pretty ready for these things.

Well, John, Matt, a chaos coordinator, a chaos specialist, thank you so much for joining us today.

Thanks for having me.

Awesome.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @_timpeako. Tweet at us, email us, argue with us, and if you like or hate what you hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.