Threat Models and Cloud Security

Security.

Hi there. Welcome to the Cloud Security Podcast. Thanks for joining us today. Your hosts here are myself, Tim Peacock, a product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed and tamed former analyst and now a member of the cloud security team You can find this podcast wherever podcasts are distributed, at our website when very soon now, we think, we'll launch it. Uh, and until then, you can follow us on Twitter, twitter.com/cloudsecpodcast. Our guest today, very special, Seth Vargo, an engineer at Google Cloud.

Awesome. Thanks, Tim. I'm super happy to be here.

Delighted to have you.

Today we have this exciting topic of threat modeling and threats in the cloud. And in my experience, I've had a lot of people who were quite confused about that. So we are really hoping Seth would shed some light on this. So how should security teams change their thinking about threats in the cloud? Are we talking different threats? Are we talking about different ways threats can affect organizations or maybe organizational dynamics? So what is special about threat thinking in the cloud?

Yeah, so in a more traditional security model, the way that traditional enterprise security works is you just wrap everything in a firewall and you're done, right? And then my central IT security team controls the Barracuda and the F5 5, and they control all ingress and all egress, and everything goes through that lovely single point of failure. Unfortunately, you can't put your firewall, you know, and your hardware appliance in front of any public cloud provider. We know customers ask and we tell them no all the time. It just doesn't work. How are you going to do that? So you have to start thinking about security differently, and it actually requires you to rethink your way of thinking. So with, you know, the traditional approach is very IP-based. which is like this IP address can talk to this other IP address and the external internet can talk to these internal resources or IP addresses. And when you move to the cloud, you have multiple network layers. So the idea of like an IP address as identifying doesn't make sense anymore. But even more so, you have like complex identity and access management so that, you know, one machine may be able to talk to another machine, but only access a subset of the resources that that machine is providing. So as a result, we have to think about who, what, and more recently, when a resource can access another resource instead of just saying, well, it's all wrapped in the firewall, so it's secure. Anything in the firewall is secure, everything outside is insecure.

Do people really ask to like ship us a Barracuda? Does that happen?

I don't know if we're allowed to specifically name vendors, but customers have definitely asked if we would install their physical security appliance in a particular data center for them.

Too funny. So, getting back to threat modeling, where and when should an organization start building their threat model for their cloud environment? Is, is a threat model like a tree? The best day to have planted one is 30 years ago. The second best day is today. How should organizations think about this?

And before he answers, I want to kind of briefly jump in and say, sometimes I've seen people confused about the definition of threat modeling versus, say, threat assessment, even sometimes threat intelligence. So, before you answer, Seth, how about you give us a quick Crisp definition of what you mean by a threat model?

Oh, I wasn't prepared for that question. In my opinion, right, a threat model is a written and documented agreement on the risks that our system is willing to take and the mitigations we have against those risks.

So when should you make that?

I actually think your analogy is really great. The best time to plant a tree is 30 years ago. The best time to build a threat model is before you've ever written the application. However, I think in reality, especially people moving to the cloud already have applications. So we have to retroactively apply that threat model. And, you know, doing it as you move to the cloud is also another great time, right? If you're on-premise today, you're already reevaluating your security controls as you move to the cloud. So why not also include your threat model in there? And when you're building that threat model, especially if you're a smaller company, it doesn't have to be very formal. It can just be like a Google Doc that is in kind of a shared drive that's easily searchable and anyone in the company can find it. Over time, you can evolve into more of like a matrix-based approach, or maybe you introduce some levels of scoring or rubrics, but you don't have to start there. You can just start by writing a couple paragraphs around, this is the risks of our system, these are the mitigations that we've taken against them, these are the unknown unknowns.

So in light of this, what I've picked from the answer is that, well, it's useful to create a threat model in the beginning of the application lifecycle. In real life, you may have an application and then the right time is now. But what if I've had this done early on and I had a big, massive application built for the on-premise world, and maybe this application was deployed on-premise, but then now I'm moving to the cloud. So what should I do with my threat model? What are the key changes to threat models after migration? Sure. If I'm thinking about it, of course, you wouldn't have to worry about some aspects like, say, physical threats. Like, this is taken care of by somebody else. And of course, now you live in a shared responsibility model. But What are, generally speaking, kind of a taxonomy or list of key changes to threat models when you migrated something big to the cloud?

Yeah, so like you said, the physical security is generally taken care of. So it really becomes a question of things like the principle of least privilege. A lot of cloud providers, for better or worse, kind of have these default permissions, which make it really easy to get started with the platform, but also make it really easy to leak that credential and then compromise the whole platform. So making sure that you know, and ideally this would be part of your original threat models, like what other services does this application need to talk to? What are the minimum set of permissions that it needs to run so that if it is compromised, an attacker can't move horizontally into the rest of the application? The other thing you need to think about, going back to perimeter-based security, is the actual network itself. So when you're in your own data center, you might have some existing layer of trust. All of your traffic might be going over HTTP or TCP.

Right.

When you move to the cloud provider, you actually really need to think about securing that network. So only talking over TLS so that if an attacker gains access, they can't just sit and watch all your network traffic. So really managing that service-to-service communication is important and understanding what that upstream service communication looks like, potentially even investing something like mutual TLS if you're trying to also authenticate the service that you're talking to.

Which, by the way, reminds me of some wisdom somebody shared with me about threat models when they said, When you're on-prem and if you make an application security mistake, you may still be okay as long as you also don't make a network security mistake. So if you make an app security mistake for an internal app, then to be compromised, you have to also make a network security mistake. In the cloud, if it's an exposed application, if you make an application security mistake, you may be done because that exposes the app to an attacker. So in that sense, it feels like a flatter threat model, I guess, right?

So I like to call this the garlic versus the onion.

Love it.

Oh boy.

Bear with me on this metaphor for a second. With traditional data center-based security, you're basically a clove of garlic. You have 2 or maybe 3 layers of security. You have the big clove, the little clove, and then the skin around it. So that's your outer perimeter firewall, some maybe network observation appliance, and then application security. When you move to the cloud, you're actually more in an onion-based approach, right? And different people own different parts of those onions, right? So the cloud provider owns things like physical security and the security of like the underlying fiber lines that connect data centers and points of presence. And the cloud provider owns like identity and access management, but you own configuring that identity and access management, right? And then from there, like the cloud provider makes sure that the network is secure, but you're responsible for making sure that the traffic you send over that network is secure. There's just many, many more layers, some of which are the cloud provider's responsibility and some of which are the customer's responsibility that you have to unravel as you move to the cloud provider. This is why I like to say traditional data centers are garlic, because there's just less layers to garlic.

So to me, this is a bit profound because my initial take was that, oh, Seth would now explain how there are more layers on-prem. So this is not what I expected. So I want to quickly drill into this. And so I got into a discussion with somebody who was trying to say that defense in depth is no longer valid in the cloud because you only have one layer identity. And we argued a little bit because it's not really true in some cases, but they also pointed out that some cases like a SaaS application, where it kind of is true, the only layer that customer has is identity. Everything else is in somebody else's hands. So how do we harmonize this? So are there more layers in the cloud or fewer layers?

So personally, I think there's more layers in the cloud, right? So ultimately identity is everything, but what that identity can do and who can configure that and the auditing and the logging and the ability to replay Like these are all somewhat unique to cloud. You generally don't see that on-prem. There's also a little bit of entanglement here that might be worth mentioning, which is typically on-prem we see a lot more monoliths. So there's a lot less inter-service communication, there's a lot less multi-regional deployments. Whereas when you move to cloud, we tend to see more, we'll say, service-oriented architectures where individual things move independently from one another. And that capability, which allows teams to move independently of one another, does introduce some complexity, like endpoint security, that I wouldn't say is unique to cloud, because you could have that same problem on-prem if you were following that pattern. So, there's this paradigm of service-oriented architectures, which introduces some security complexity and additional layers of the onion that is tightly coupled with cloud, or at least correlated to cloud.

I like that. And Anton's question was much better than my follow-up, which was simply going to be making the quip, That with a move away from garlic data centers, we are now much more exposed to vampires as a risk in cloud computing.

See, I'm glad you threw that in because I feel far less guilty now about what I was going to say after my last thing, which was, remember, my statistics degree is obligated to remind you that causation and correlation are different things. For example, the number of shark attacks each year is correlated with the number of movies that Nicolas Cage has starred in.

That's remarkable. Who would have thought that? So specifically when it comes to identity, securely storing credentials, lateral movement, what's a user to do? What would you recommend to the, you know, the top things that security team ought to be doing either on the monitoring side or setting up secure guardrails for teams?

Yeah, so the first thing you need to do is you need to set aside some budget for incident response and detection. All too often I talk to customers and potential customers and they're all like, okay, we're going to invest 110% in prevention. And I don't care how good you are or what cloud provider you have, there are going to be unknown unknowns, right? Things like Spectre and CPU-level vulnerabilities that came out of nowhere, they're affecting everyone unilaterally. And if you invested all of your security budget, I'm not talking monetary budget, I'm talking, you know, human budget and engineering budget in prevention, and you haven't put any effort into response or detection, you're basically screwed.

So is it kind of a 1990s thinking, right? Come on, this is 2021, right? This feels very 1990s. Do people still do that?

Definitely. Customers do this all the time to the point where they're like, well, how much? And I generally say like, you should budget at least 25% of your engineering budget to go towards analysis, anomaly detection, response, and prevention. The remaining 75% should go towards detection. And honestly, it varies by customer and vertical, but that's kind of the general rule that I go with.

That's very interesting because I work on a team that does detection products and we think all the time, well, what's the right breakdown of engineering investment between, you know, our left-hand vulnerability misconfig stuff and our right-hand detection stuff? And right now it's probably closer to 50/50 for our products in terms of where we're investing time.

Let's take a concrete example. Let's look at like CVE scanning for a moment, right? So if you're looking for common vulnerabilities in code and supply chain, that is a point-in-time scan. So, if right now I ran any industry standard CVE checker, it would look at my code, use some static analysis and say, oh, okay, I don't see any CVEs that currently exist today on April 19th in your code. I deploy that code to production and then I go on vacation for 6 weeks. The world doesn't stop though. Like, new CVEs may exist that are disclosed after I've run my scan that are actually in my application. So this is where, you know, we talk a lot in the operations space about like continuous delivery, continuous deployment, continuous testing. We also need continuous scanning and continuous security. We need CS, not to be confused with computer science, because security is a moving target. It's never done, right? A lot of customers I talk to is like, oh, I just need to do this and then I'm done. If you're done, then why do you have all this staff? Like, what are you going to do with them when you're quote unquote done? It's constantly evolving. And this is why I like to call it security engineering. Because it truly is an engineering discipline.

I like that. I wanted to roll back and talk about hybrid cloud because we've talked about on-prem to cloud. What's a user to do in a hybrid cloud world where there's different identity models in each cloud, different tooling available? How do they possibly make sense of that world?

So don't convert an identity management problem to a secrets management problem is my number one piece of advice.

What does that mean?

One of the biggest mistakes that I see customers making is they say, I need to talk from cloud provider 1 to cloud provider 2, or vice versa. So let's take a concrete example. I need to download an object that is stored in Google Cloud Storage from Microsoft Azure. The way that most customers would do that today is they would go create a service account, create a service account key, give it permissions to talk to that bucket in Cloud Storage, and then stick that in like AWS Secrets Manager, or run their workload on Azure and call GCS. The Problem with that though is you have exchanged an identity problem, which was the identity of the workload on Microsoft Azure talking to Cloud Storage, with a secrets management problem. You now have this password, it's a service account key, but think of it like a password that you now have to manage, you have to rotate, you have to secure, and you have to audit. That's the biggest mistake I see customers making. The good news is that most cloud providers have a concept of federated identity today, typically based on OIDC. This allows you to delegate trust to another cloud provider, requires a little bit of configuration. On Google Cloud, it's called Federated Workload Identity. And in this way, you don't have to download that service account key file. You can just set some configuration that says, you know, this Azure instance ID or this AWS ARN has permission to talk to my Cloud Storage bucket, or more accurately, has permission to impersonate a service account that has permission to talk to my Cloud Storage bucket. You've kept the problem at the identity layer. You haven't dropped down to the secrets management layer. All too often as humans, we think, oh, let me just download that password or the equivalent of a password, and then I'll stick it in my on-prem or on my other cloud provider, not realizing that you're just exchanging one problem for another. And there's actually a much better way to do that.

But also to a bigger problem sometimes, because from the public breaches, we do see how insidious credentials spread all over the code is. Credentials on GitHub, credentials in this place, that place. PHP files, I don't know, with passwords. This feels very 1990s, yet it happens in 2020s, right?

Well, exactly. And, you know, circling back on what we were talking about earlier, if I use the identity-based approach, when I'm looking at my audit logs, I see the exact identity that invoked the API. But if I leak my service account key on GitHub or GitLab or some S3 bucket, all of the requests are coming from that same service account. And I might be able to get some granularity by like IP address, but I don't have any mapping back to the identity because the service account is the identity.

This is kind of depressing. We need to pick up on something positive. So how about people who are successfully leading the cloud migrations? I mean, there are organizations that are quite successful getting to the cloud securely. So how should people who are in those situations, how they should help their colleagues think about security in the cloud? Of course, we talk about different threats. We talk about changes, how we deal with threats. We talk about changes in balance between, say, detection and prevention. So what's your best advice to people who are in the middle of migration? How do they manage this up? How do they teach others in the organization? How do we spread this knowledge so that more organizations move to the cloud securely?

The only thing better than like a document or a policy or a training is code. So if you can automate any of this using things like, you know, Terraform or Deployment Manager, where you have a factory of security best practices that your developers and your sister teams can just grab and go, and you know that that's been reviewed and approved by the security and policy teams, golden, because you actually don't have to teach anything at that point. You have captured those, what I like to call in code, or you've codified those best practices. That's not always possible. There's obviously a lot of gray areas in security. So training is really, really important. Continuous training is really important. And having a place where developers and infrastructure engineers can go to office hours and ask their questions on things like a Google Group, Or scheduled time so that they can get consultations, right? Internally at Google, we have the ability to like go ask for security consultation or security review. And we generally encourage that our customers set up a similar group in their organization that can be responsible for answering these types of questions and providing recommendations. And then finally, you know, the absolute last thing is third-party training and certification. So if you are on a cloud provider, potentially looking at some of the courses that they may offer as a first-party cloud provider or as Like third-party, like Coursera and various like third-party courses that offer training and certification and sponsoring that as a company to help your employees grow their skill set and become aware of some of the challenges that they might have when moving to the cloud securely.

As a final question, getting back to the original theme of threat modeling, when is an organization done with cloud security planning? When are they done?

Either IPO day— no, I'm kidding. You know, you're never really done with anything in security. It's constantly a moving target. This is why it's important to celebrate milestones.

Hmm.

But you're never done. It's constantly a moving target. You're never going to hit it. That can be daunting. So we have to celebrate the wins that we do have, but you're never done threat modeling. The space is constantly evolving. I highly doubt, you know, 5 years ago any threat model included, oh, there's a CPU-level vulnerability that allows like anyone access to all of our data. But I guarantee you that today it does. And this is just one example of how the space is evolving and how we constantly need to be thinking about it. So you can take a vacation and you can take a break, but you're never done.

I like that. That's actually one of the things that I love about working in this industry is that we're never done. It's always changing. We're always learning new things. That's one of the reasons I like the podcast too, is we get to meet all these interesting guests who have different perspectives. So with that, we're at time today. Listeners, thank you so much for listening. Seth, I can't thank you enough for joining us today. You can find this podcast on Google Podcasts or wherever podcasts are sold. You can follow us on Twitter at twitter.com/cloudsecpodcast. My co-host is @Anton_Chuvakin and I am @_TimPeacock. Tweet at us, email us, argue with us. If we like or hate what we hear from you, we might even invite you on the next episode, which I promise is more fun than arguing on the internet. So with that, we will see you on the next Cloud Security Podcast episode.

Security.

Security.