EP193 Inherited a Cloud? Now What? How Do I Secure It?

Secure running.

Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google PsychOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

And today we are talking about the subject of doing cloud security late. Our guests today are Taylor Lehmann, Director at Office of the CISO, Google Cloud, and Luis Orena, Cloud Security Architect at Google Cloud. Welcome to the show.

Hey, Anton, it's great to see you again. Thanks for inviting us back. I can't wait for this conversation.

Perfect. So, this episode was born out of a blog or set of guidance we're writing, and it's for a scenario which is surprisingly common, even if very painful. It's when a security team inherits a cloud environment. For example, a cloud environment that was set up by some other team where security wasn't involved. So we are thinking about this as doing security late. It's not a good scenario, of course, but it does happen. So how does this late involvement typically impact the organization's security posture, and what are the immediate risks you face? Maybe we'll start with Taylor.

Well, you know, this situation comes up a lot, Anton. I can remember times before working at Google where I was in these really tough situations where You know, we had just gotten done implementing a security control like multifactor or something, or that we had just gotten a new merger with another company, or even like we had an incident and we were responding to the incident and we just found something new that wasn't there before that nobody knew about. And everybody goes, oh my gosh, what is that thing? And that thing oftentimes was a cloud account or a cloud project that lived outside of what the organization thought its scope was. And so, you know, going with the example of a merger and acquisition, you know, pretty much all of those new workloads coming over from that, if you're the acquiring entity, and all of those workloads that you just acquired kind of fit this definition of, oh my gosh, what is this new thing, right? I have no history. of where it came from. I don't know who secured it, what the policies were, what tools are in the case. And that's a relatively common one. We see it all the time, especially we see this when teams adopt new clouds for the first time, right? We see this when we go from, hey, we're experimenting with cloud, to, hey, we're now in production, right? And so that team that created that thing, which had no business talking to security, all of a sudden wants to thrust their thing. So we see this happen all the time. But there is a new project or a new account that we now need to secure. And it's like an oh-my-gosh moment. Like, where do we start? What do we do? Can we trust it? Can we not? Does it have the security primitives built in? Does it not? So there's all these questions that security professionals have to sort of sort through to get to what is that first step.

That makes sense. And I think it's the unknown nature of the risks is, I guess, what scares us the most. It's not like there are known risks every time. It can be different every time, right? It's the unknown nature is what makes it kind of a head-scratcher.

It's always different in the sense that where it came from and how it got there, those stories really never really line up. But what is common is this feeling of uncertainty of what to do.

So, Luis, you probably saw customer cases, and again, no names, please. How do people think about the risks of such a situation?

Yeah, I've worked with plenty of customers who were in the same exact situation Taylor described. but also others that were effectively startups. And they were growing so quickly that they were just focused on achieving that product-market fit, finding their customers. And as they expanded, the risks came out. And it wasn't necessarily anything like a breach or an incident. Like, things were running smoothly. They were growing. But they had grown to such a point where now they faced market risks.

Mm-hmm.

So in order to sell to the healthcare industry, to financial services, to governments, these customers realized, We need to get security in order, right? We haven't suffered anything yet, but we need to do this because right now we're facing that kind of market risk or product-market fit for a new industry. And security was really the driver that led them to me, at which point we uncovered plenty of technical risks, usually in the identity space. But I think that the business outcome and the risks to market expansion is something that I also see quite often, especially with organizations that are sort of newer to the overall market landscape and not deeply established entities.

So that means that maybe there was a brief period in their history when essentially not doing security was seen as the right call, but they've grown, and maybe they started addressing regulated clients or deal with regulated data, and that approach is not going to work anymore. So they have to do it, and suddenly they have environments grown without security, which they themselves have to secure.

Yep, absolutely.

So, one thing that came up, and of course, when facing this tough question, I've done my usual social media polling. And a fair number, probably the most popular answer from this, sort of the most popular single answer from a lot of experts, was that this environment should be destroyed. Burn it with fire, nuke it from orbit, remove it, rebuild it from scratch. And of course, in the Loïs's scenario, where it was the startup entire company, that's a, to put it mildly, non-starter. So, what is your take when people like security purists show up and say, oh, you inherited this? No, no, no, no, no. You have to destroy it. You can't do it. So, what is your reaction to this? How do you deal with this?

I mean, I wish them luck and good job in your career. It's now over. It was wonderful to watch your ascendancy to this position where your opinion was asked, but you answered it the wrong way. Look, like, you know, we're here to help manage risk. That's a security person's job. And so, we must assess what the risks are and make appropriate calls. I think in the situation that you're proposing, so let's just be clear. Let's put ourselves in this scenario. You know, one of us has been asked to provide input on This new environment that we've just discovered, what do we do, right? And the answer can never be burn it down, but there are some steps that one can go through and should go through to understand the risk of the environment and identify what steps to take. Now, Louis, very succinctly, brilliantly, I might add, pointed out identity is usually the hard part. And that's usually where we ask folks to start is look at— Who's getting into the environment? How is that happening? And making sure that we completely understand the identity plane first. That's usually the first place where we have to start making changes, as well. So in some of the guidance we're working on, you know, in this situation, we always encourage people to get access to the environment as an organization administrator. That's the first thing you need to do. That way, you have the ability to audit or inspect everything else in the environment, but... And most importantly, you can start to take control of the environment, which is really the thing that needs to happen first. Burning something down indiscriminately is a bad idea. We need to get control of the environment. And at Google, you know, there's a couple different ways to do that, some conventional, some not. We find people in different situations all the time. Hey, the guy or gal who built that thing doesn't work here anymore. What do we do? Well, you know, at Google, you call cloud support, and we have a way of helping you get through that. There's also times where it's a private account or a personal account, and we want to make it a corporate account, right? There's steps you go through that add a project to an organization in the event that it's not there. So these are the first kind of steps you sort of need to work through to sort of like transfer liability correctly, to gain control of the account correctly, to then conduct an effective risk assessment. But you got to start with getting access to the environment and assessing, hey, where are the landmines? What do we need to do first? And usually, that's about getting control and understanding when and how to transfer liability. And then, of course, you know, you start to look at the workloads and understand their risk, what data is present, and you work your way down. Now, you might ultimately decide quickly that, hey, it's just not a high-risk environment— or excuse me, a high-value environment. And sure, this is more work than we need to do. So yeah, of course, you can hit the button and get a new one. But that is very rarely the case. And once you get access to the environment, the other general advice that we give folks is treat it like an incident response situation. Execute your incident response playbook, which basically tells you to figure out what you've got, figure out what access you have, figure out what's valuable about the environment, figure out who could be in the environment, find a way to understand and sort of evict things that you don't want, return control, do it again until you're confident to your already internal standards that the environment is safe. Then we can talk about steps after that.

Yeah, and that's perfectly stated. And I just want to say the burn it down approach, it can happen. I don't recommend it. And the way I like to think about—

Wait, wait, you don't even recommend it? Not only you have to accept that it's impossible, but you actually wouldn't recommend it? Sorry for interrupting you, but this is an interesting point because I thought from a purist point of view, that's recommended. It's just you can't do it. But you are saying it's not recommended.

Why?

Why not?

Yeah, I wouldn't necessarily recommend it because You already have stuff that's running there.

Mm-hmm.

Things are working. So why would you tear— let's use the analogy of a house. You're going to buy a house, you inherited a house. Why would you tear the house down when you can instead start inspecting it, keep the parts that work, and fix the parts that don't? So in these situations, it's not like you're just having like a test account and everything is immutable and it doesn't matter. The things in the environment do matter. And so we want to make sure that the things that are running continue to run, but we're mitigating and reducing the risks over time instead of just saying, oh no, the hell with this, we're not going to keep this environment, we're just going to start over. How do you handle the traffic? How do you handle your customers? What are you doing in the intermediary point? So those are the reasons I don't recommend it.

Interesting. So that makes sense because ultimately somebody would be on the hook to making it work as well as before, but also be secure.

I mean, imagine you're a customer and some rogue security purist came in and said, ah, your environment, we don't know it. Boom, it's gone. Like, and you're out. Like, sorry, businesses are in the business of staying up and running.

Doing business.

Availability is critical, right? We have to keep services going. So, if you ever hear somebody come in and say, let's just wipe it away, which does happen quite a bit, you might want to find somebody to get advice from that isn't that person.

That makes sense. And what makes me nervous sometimes is that there was also sometimes the opposite advice when people show up and say, well, business demands that you don't touch anything, and if you bring your security thingies, then they would mess. So, like, there's an opposite one. And of course, it's a perfect tailored question because you were a CISO. So, how would you deal with that situation when people say, no, no, no, don't touch anything, but make it secure, which of course is crazy?

But yeah, I mean, the snarky answer I gave in the chat earlier was immediately quit and then write 2 letters. Right? Like, you know, and then threaten to put both in the mail at the same time. Look, I honestly think this is almost always the situation that one will find themselves in. It's very rare that you would just discover this and, as the security person, being given, like, full range of decision-making authority of what to do with this. Oftentimes, these are time-compressed situations. Oftentimes, you have to get this done within a, you know, within a certain level of quality. And, of course, no outages are allowed or whatever, right? You know, these are really tough situations. I think the right answer to this question is really the answer sits in how you respond to similar incidents in the past and what your playbook tells you to do. It's not something you're coming up with at the moment, right? It's something that you've discussed with the people who are pushing these environments on you ahead of time to say, look, when this happens, this is what we're going to do. This is how long it's going to take. These are the steps we're going to take. And if, hey, if there are risks associated with bringing this thing on under our liability chain, that those— how to handle that has already been prescribed and we've got a process for it.

Right.

That's like the ideal situation, right? But if you can't get that, you know, we recommend organizations first look at ways to isolate those workloads, like isolate through identity, isolate through network, isolate through service restrictions, but like create isolation that basically prevents any badness in those environments from spreading to other environments. And that needs to be done at all layers of the stack, you know? And so, like I said, identity, network, service restrictions, Those are the places to start. And then within those areas, it's paring things down to least of whatever allows. So least access, least networking, least services. So it's one thing to isolate a service, but it's another thing to just, like, remove a service that's there that you don't need. And so as you work your way down to isolation, those are typically things that are easier to do than, say, rebuilding an environment or finding and patching vulnerabilities and/or restructuring something for cloud. Well, those are all things that are probably better discussed after.

Mm-hmm.

You've sort of— what do you say in medicine? After we stabilize the patient.

Yeah.

After we stabilize the patient, we can then worry about getting it healthy. But, isolation is the next step. Once you get access, isolation. Once you get access, running your IR plan, making sure that there's no badness. And then, you could do isolation at the same time, but isolate, isolate, isolate all the way down to the thing where it's like, okay, there's nothing else that could go wrong here. And then, you talk about rebuilding. then we can talk about whatever the future is, rebuilding or something else. And I agree, if you're jumping right to rebuild, like, you're doing something wrong.

So IAM was mentioned a few times, and I think that taking over the environment and getting access is an obvious first step that I'm sure people would miss, but IAM shows up soon thereafter. And so I wanted to ask Luis about this classic advice about MFA when everybody's just like spouts on the internet, hey, add MFA. So should we also say, Add MFA everywhere, some environments. I mean, we sort of hear about those cases when something blows up when you add MFA, so people are a little nervous. So what's your take? Why not just say add MFA everywhere? What may break, Luis?

Yeah, I think you should add it, but it has to be carefully done, right? You want to ensure that all of your personnel can actually implement that second factor. Otherwise, you're just risking locking them out, and then you have a different problem on your hands, which is, do I now remove MFA and deal with that pain, or do I want to make sure that my people and my teams can continue having our service running. So my recommendation is start adding MFA, but do so in a phased approach, including, like, talking to your communities and your teams and having a communication strategy so that they can go ahead and understand what MFA is, why we need to use it, whether it's optional, whether it's required, And if it's required, what is the plan for getting everyone on board? Now, the other part of that is there's many forms of second factors. So if you're thinking about something like security keys, which I think is one of the stronger methods, you also need to think about when your personnel are going to physically receive those keys. Otherwise, again, you're going to risk locking them out if the deadline passes and the mail person hasn't showed up with that security key. Right? So it's all about having a plan, having that communication strategy, and then doing a phased rollout so that essentially your administrators and those with access to your most sensitive data have MFA enabled as quickly as possible. And then from then on, you can roll out the MFA across your personnel based on their data access levels their system access levels and essentially continue to take on that risk-based approach and stabilizing the patient, like Taylor said.

That makes sense. And I think that any other MFA risks, Taylor, anything to add here?

Yeah, I mean, there's plenty of risks. The one I'd say is overreliance on any single control measure to fix everything, right? Like some people, yeah, we're good. It's like, nope. MFA is like a high-level abstraction for isolation. It just isolates an environment on the identity plane from, like, the rest of the internet. But, like, it doesn't answer, hey, we've got super overprivileged users, right? And we've got MFA inside use cases we need to think through, too. So the only mistake is thinking you're done.

But overprivileged users are nasty, as well, right? It's, like, a very common way when the roles and users and identities just get too much privilege. And we, of course, say address this, but isn't it, like, really tricky?

I mean, if you're using anything other than GCP, yes, it's very hard.

Good one.

Yes. I mean, well, I'm kidding. I mean, like, one of the really cool, like, unstated features that we have is this thing called IAM Recommender, which is, you know, we bring machine learning to everything, including how to secure your workloads even better. IAM Recommender, you know, basically looks at how identities are being used and what privileges are being used in a given organization. And when you turn it on, it's going to go through and spit out a bunch of stuff that says, hey, you know, Louis has got all this access that he doesn't ever use, ever. So please turn these off, and you will give Louis exactly what he needs to do his job, but not, you know, domain admin or the equivalent of in cloud. And so, like, you know, we talk about, like, establishing isolation and then driving it even further and deeper down. Tools like IAM Recommender play a big role. And, you know, it's the same kind of activity and the same tooling for networks, right? For services, those are a little easier. You can just use service controls. But I'll tell you, like, Having worked on some very large electronic health record systems in the past, like, it can be really challenging to get down to that least privilege without these types of tools. And so, you know, hard to argue that people are going to pick their cloud based on the availability of their tools. But if you're a security professional, you've gotten burned by this before, you know how important these things are. And they do become very, very handy, especially when you find yourself in a situation where you just discover a new workload and you're like, oh, well, It's like, this is one of the first things you do.

So one other thing that comes up, I think you, Taylor, brought up the point that treat it as an incident. But in Luis's scenario, when it was a startup that maybe ran without much security for a while, for them, it's business as usual. So one suggestion that came up is to kind of run a compromise assessment on the new environment. So I'm almost torn between this because it sounds like good advice. What if it's already compromised? But at the same time, you've been running your business. on this for some time, so you doubt it. So how do you approach doing compromise assessment or not doing it? Like, it costs money, right? At least Mandiant's one does, the best. Another marketing hit here. But how do you think about this? Would you always run it? Would you sometimes run it? Let's start with Luis and then get to Taylor.

Yeah, I think in those particular situations, I would actually recommend it because—

Oh.

Yeah, I would. Here's the thing. If you're missing security controls, you likely are missing the detective measures that will let you know whether you're compromised. And so that's where I think the compromise assessment could be really valuable because imagine you're going through all the effort, all the expense of securing this environment, and the intruder's already there, and they're seeing everything that you're doing. So now you think you're secure while you actually have an intruder there. The assessments do cost money, but as you think about that cost and then you compare it with the cost of investing in improving your security while this intruder is in an environment, then having a consequence because of the intruder, suffering through that reputational and regulatory damage, and then paying for incident response services, I think it is worth the investment to just go through the compromise assessment, see where you are, And then harden your environment because the results of that compromise assessment will also let you know exactly where you need to prioritize your efforts. So I think it's the scenario where you're actually going to have the most return on your investment.

Yeah, I would just— like, compromise assessment, we're talking about like it's a separate thing. Compromise assessment's probably already built in your playbook for IR, right? Like, you got to kind of know, kind of come in with the assumption that there has been a compromise. In that case, the assumption is pretty strong, right? But as part of that assessment, like, I'm saying this for all the humans in the world that are purest security people, we must establish what is the scope of the environment and specifically the things we care about the most. And you can't do good compromise, you can't do good IR unless you know what the asset is that's being attacked that you want to protect, right? Like, before even the compromise assessment, or maybe it's part of it, right? And I'm not saying go run a bunch of scans on a bunch of stuff, but I am saying we need to establish a clear understanding of what is at risk here and why do we care, which ultimately drives you towards an inventorying exercise each and every time. And then, you know, of course, getting the compromise assessment up and going is going to force you to learn how to deliver software into that environment and deliver access into that environment. So that's why I come back to, you got to come and look at your IR plan and execute your IR plan. But your IR plan needs to contemplate these things to be useful in these situations. But it's why that's a great tool.

Mm-hmm.

to use when you find yourself in a situation where you've been shipped this wonderful environment or you've just discovered it and, you know, the party's over. You're like, yeah, we found it. Now what do we do? You know, it's that IR plan. But that compromise assessment step is key. It comes after a couple other really important things.

Okay.

So that means that for high-stakes situations, when it's literally your entire business, you're more motivated to do a compromise assessment even though it costs money. Compared to just, you know, making an API call to see what the assets are or something more lightweight. So, it seems like you want to have a way to give it a relatively clean bill of health, right? Like, that's the idea, it sounds like.

Yeah.

Okay. That makes sense. So, we're getting close to the end. So, I wanted to ask a somewhat tricky question. So, when you inherit an environment, whether it's through M&A or whether it's shadow IT is becoming official, so our security team may have Well, other stuff to do. So, as a security leader, how do you balance securing what you had to secure anyway and this inherited thing that just got plopped on your plate? So, how do you make trade-offs between working on your existing stack and the new one? And please don't say it's all high priority because, like—

I said you gotta— it's— you write 2 letters, Anton, not 3.

Yeah, but it's a depressing thing. Give me a positive Taylor tip.

What do you do? I mean, look, this is like the day in the life of anybody working in security. Like, I wish I knew the answer to this question, I'd be wealthy. How do you balance your team's priorities? You know, I think some people would say you should have some— what do we call this— unplanned work time built into your schedule. Slack, right? The optimal situation you might find yourself in is that you know this stuff is coming, That's not always the case. It certainly doesn't track to the examples we've been using today. So, I won't put that out there. But again, this is a surprise, right? That assessment needs to come up and tell you that these risks outweigh the risks that you're currently mitigating in your environment. We all like to say, well, let's go hire a team who can go deal with this for us.

Yeah.

And so, a lot of people do do that, right? Not to sort of plug again, but like, that is very oftentimes the tool that is used is to say, look, this is how we deal with surprises. We need to burst resources. We have people we burst with. So it's good to have your vendor deck built out. It's good to have simple statements of works that can be issued quickly that get these people in the environments. This is like, again, going back to you're practicing your IR plan. You're going to need to be good at that anyway, you know, in the event you have an incident, right? So it's leveraging and flexing those channels on how do you get work done quickly? How do you bring people in with the right skills who know what to do quickly? That's something you practice. That's not a security control or technology. But, you know, it also helps that what I've seen some organizations do is that they have staging areas built out, right? They can bring an environment into a staging area, isolate it quickly, and then buy themselves some time before they migrate it into a production environment. And it's really helpful, especially if you're on cloud where you can spin these secure foundations up, bring things in, fix them, move them into the permanent home, and then burn the secure foundation down, all without having necessarily to deploy much new tech. Again, that's a little bit more sophisticated, and obviously those environments need to And change based on what you see. But there's different ways and tactics to deal with the immediacy of some of these events. And I really do think it comes down to traditional things that a lot of orgs don't do today but should do more, which is, right, practice your IR plan, conduct a stress test, do a tabletop, just see how broken all your stuff is all the time. You're always going to find stuff that you can do better. And I really like this scenario as maybe a use case, right? Like, here's something we should tabletop. Let's discover a new environment in our cloud. Let's discover a new environment on-prem. Let's discover a new environment in 2 other clouds connected to some other cloud. Let's walk through how that might work today and then just see, you know, let's just see what we need to do. And then, you know, I think those exercises really do make it easier for you to make arguments like, okay, cool, we need to be able to move quickly. How do we do that? And then you do it.

I think that makes sense. And I think that while I expected some magical no work, no planning kind of answer. I think that no security work and no planning is what led to the situation. So, it's unlikely that just by magic or by pushing the button, security, it can be done. It's kind of back to following the steps and a non-trivial number of steps at that.

You know, you can use Gemini too. That'll tell you the answer.

It probably would be a similar answer because it's at this point becoming common wisdom.

Right.

So, I need to get to our traditional closing questions. And I think it's If you obsessively listen to the podcast, as hopefully many of the listeners do, traditional closing questions are one tip on the topic and one reading resource. It can be a book, it can be anything. So, Luis.

Yeah, I was thinking about this, and because we've gone through so many different topics, but we haven't talked about threat modeling. So, my tip is really focus on threat modeling, go as deep as you can, And of course, anytime we're talking about threat modeling, we have to recommend Threat Modeling: Designing for Security by Adam Szostak.

Oh, Szostak getting a shout-out. This is like the greatest Friday of recent history.

And you know what? Shout-out to Taylor and Bill Reid for introducing me to Adam back in January. So, that's a little nugget there.

He's one of the only humans on Earth who can make threat modeling interesting.

Yeah.

You can use Star Wars analogies like he does all the time. You can make anything interesting. I really like the threat modeling one, Luis. I'm not sure I can outperform that one or even come close to it. Mine is a little bit more boring, and it's just general advice, like, don't wait for things to happen. Don't wait until you find yourself in this situation, right? Like, envision yourself in this situation and practice what you're going to do. Give yourself a little bit of time when you have it to think through, how is this going to work best for my organization? Because there's no playbook for this. There's, like, there's literally no playbook. There's no situation that's going to be the same. And I remember having this exact issue during the 2020 Olympics. You know, the situations, you'll never find yourself in situations where this is not like an urgent priority with tons of other stuff going on at the same time. So, practice, practice, practice. Put your plans down. Use automation. Use, you know, infrastructure as code to move quickly. But, you know, make those decisions based on your lived experiences and not what you hope to happen when things happen. They don't work the way you think they do. The one book I would recommend, or the reading resource, is a little kind of extensive, again, and not as fun as Adam Shostak, is, you know, I really like the writing of a doctor named Atul Gawande, who wrote a book called The Checklist Manifesto. And it talks about the importance of bringing structure to your work that effectively every step is very data-driven, and it's followed closely and updated when needed, but it's ultimately about consistency. Especially in times where the stress is high, having resources next to you that will tell you what to do in, in times where it may not even be possible to think are really, really helpful and important. So I encourage everyone to go read this. It's a book about medicine, but it applies really effectively to any real situation where stress is high. And being a healthcare guy, I value these insights greatly. I think in this case, it talks about how checklists help save human life. I think there's a lot we can learn about cybersecurity from it.

Perfect. Thank you very much for being on the show. This is hugely useful. So hopefully our listeners who find themselves in that situation, which at this point is many of them, will have a little more pointers. And of course, expect our blog to come out soon.

Thank you.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Secure running. Secure running.