EP34 Instrumenting Modern Application Stack for Detection and Response

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever podcasts are sold, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is one of the more cloudy episodes, wouldn't you say?

Indeed, indeed. And to me, it's also the one that highlights some of the things that really challenge some of the defenders, some of the blue teams, of the SOCs, because we are talking about doing threat detection in a thoroughly modern stack, in a cloud-native stack. Think microservices, think containers, think functions, think about other stuff that just doesn't fit into the servers and boxes and devices and even VMs model. I've seen a lot of defenders challenged by that.

I love asking users to think with cloud. You know, like when you play Portal, you really wrap your head around a different way of seeing the world. When you move to cloud, it's really tempting to just lift and shift your thinking, forklift a bunch of servers into some data center you don't really own, and go on with your life. But if you're going to play with things like Cloud Functions and serverless container deployment and everything else, you really do have to think different. And this is a great example, I think, of a deep conversation with a security expert who's spent a lot of time thinking different about these problems.

And the only other thing to note is that people who are working for more traditional organizations would listen to the episode and they would wanna take some of the lessons on board. But do keep in mind that the guest today is in fact a cloud native. So his lessons are perfectly applicable to cloud native environments pretty much out of the box. But you may have to work a little bit harder if you do have this environment in transition from old world to new world.

I was chatting with my director one day and I proposed doing something that he said was not in our organization's DNA. And I told him, no, no, No, Jess, that's why we're here. We're here to CRISPR this company. This is, I think, for a lot of security teams, an opportunity where they're going to have to CRISPR their security team's existing thinking. So perhaps with that, let's welcome our guest.

Today, our guest is Matt Svensson, a senior security engineer at BetterCloud. And the focus today is monitoring, well, for security issues of modern security stack. Think serverless, think containers. So to me, I've encountered many people being challenged with monitoring things other than classic servers, other than on-premise stuff. So, let's talk about this new environment. So, Matt, what do we define as modern stack, and what are the challenges with monitoring this?

Yeah, I guess I'd say modern stack where you are not running your workloads on a compute instance that you own and that you manage. So, running those workloads, whether it's something like a Cloud Function, Lambda function, where you're just running your scripts, or like you mentioned, throwing a Docker container into Kubernetes that You don't really own all that architecture.

So this really is a case of the cloud is just someone else's computer thrown on its head because there's no computer at all.

Yeah, exactly. Turn that up to 100% and yeah, that's what you got.

So why is detection here hard?

One of the first things is coming from kind of your traditional network security mindset, you don't have your taps and spans. That's like the feel-good for a security person. You attach that tap and span, you hook up an IDS, and you sit back and sigh relief that I've got security monitoring going. But you can't do that in a lot of these environments.

And why is that?

What about application logs? Can you use the old approach here?

Sometimes when you look at stuff like a cloud function, the developer really has a ton of power to say what type of response code you send. So you're not getting a 403 response code, a 404, you're getting whatever that developer says. and whatever they decide to output in those logs. So it's a lot different than looking at NGINX logs, Apache logs. You don't have that same telemetry, at least not telemetry that you're expecting.

Got it. So I've heard some people say things like watch for spikes in Cloud Function invocations. Does that work?

It works on a subset of a subset of issues. If you have a workload that you know is invoked once a day, Yeah, you can look for it being invoked twice a day, 3 times a day, but a lot of these workloads are done in a serverless type architecture intentionally because you know it's going to be spiky stuff and you don't want to run a VM on it, in addition to it being simple to do. So if you're looking for those spikes, well, yeah, you probably are inherently architecting it that way because it is spiky.

But wait a second, so network monitoring is not quite good, Traditional log analysis, where the analysis is done by the vendor, is not so good. There's nowhere to put an agent, so EDR and friends, not so good. Aren't you running out of ideas here? Like, what are we going to do?

I'm a huge Security Weekly fan, so I'm going to kind of echo some of Matt Alderman's trifecta of what he looks at, which is app user data. So instead of looking at what logs are coming out of it, You're looking at things like doing code reviews, some of the standard things of looking at dependencies. But then when you look at the user and data side, you're looking at the workload identities, what's running as the service account on that workload. You're looking at the effects that that workload has. What is it reaching out and touching? And what can you glean from those effects to see if something's going on?

So in a way, identities are the new endpoints and API calls are the new network taps.

Yep, absolutely.

That's a familiar theme, right? We had a few episodes where people would be selling this identity-centric view where you're not really monitoring logs, you're monitoring identity activities, which to me is a subtle difference. You're not picking standard logs from, well, a web server because, hey, I ain't got no web server here, but we have identity activities, right? So that to me is an interesting point because I was about to say, ah, it's log analysis, but it's not really log analysis. It's not a traditional log analysis where you know the patterns in advance.

No, it definitely isn't, because this gets into things like, I need to have a distinct service account for this one workload that has a very fine set of permissions. That workload is supposed to reach out and touch a database. Well, all it needs is database read permissions. So it's very different than looking at logs and saying, I saw something going and reaching out with this password. I saw a 403 error. I saw a permission denied. It's about ensuring that you are doing those least privileged things with a service account that's only tied to that one function.

So that sounds like a preventive best practice more than a detective control. What are the detective things we should be looking for here? Is it like we're saying, you look at what the identities are up to, you look at what APIs they're calling?

Yeah, so in any of these systems, you will have an ability to look at the workload, that service account, APIs that it's not authorized to call. So you are still kind of like Anton said, you're still doing some log analysis type work where you're looking at that service account making calls that it's not authorized to do. So that would be a version of a detection that you can do for this.

But I want to step back for a second because we started talking about a couple of scenarios, but like, wouldn't they be an example of a particular threat model? So how would we even threat model? Would we say, Somebody wants to get in a server. Oh, wait a second. What do you mean by somebody? Well, identity. What do you mean a server? Okay, we don't have a good answer. So how do you approach threat modeling here? Is it about service accounts? Is it about something else? Talk us through a bit of threat model thinking, please.

The first thing that I would look at would be what is the exposure of that workload? So is it a Cloud Function, a Lambda function that is publicly accessible that anyone can make an HTTP call to it? Is it one that is only accessible by specific service accounts? You can set up an identity to be able to access that workload, and only that identity can. So some of your threat modeling of who can access it, what's the exposure of that workload. From there, kind of going back to the what impacts can it have, you're looking at what does that thing do. Is that cloud function, is that workload doing a sync of data, and if someone invokes it, they can't do anything but cause a sync of those data anyway, or is it doing database reads and writes? So looking at what that thing is doing and what it touches internal to your systems, because that's where you'll be able to do a lot of the detections as well, is looking at impacts from what that workload is doing.

Interesting.

Sounds like we are talking a lot about the data confidentiality impact, like somebody taking data out then. Or would there be presumably some kind of DoS risks as well? Like perhaps you would look for activities being increasing not by a factor of 2, but by a factor of 1,000 millions, whatever, a billion, so that you have a DoS or maybe you have some kind of a malfunctioning application doing something. Like would you have that in your threat model or not?

Yeah, I think it needs to have a place. I did some testing looking across S3 buckets. If I do that in a compute instance, it costs me like $10 to do it. I do it in Cloud Functions, and that same thing costs $500 to do because it can be expensive. So you do need to look at it, if nothing else, then not just for security because of that operational impact. So it does need to be a part of that model. I don't think that is the end-all be-all of looking for spiky activity. It's a factor that you need to be able to look at though.

One of the things we've been talking a lot about here is data confidentiality. Are there threats to integrity of the cloud functions and how would we monitor for that? You know, what if an adversary doesn't wanna steal your data, He wants to tamper with how your application works. What are we looking for there in a Cloud Function world?

If you are looking at how can someone modify your Cloud Function, the hope would be that you're following some of the best practices, if you will, of being sure that you're only pulling in code that you know is safe. So pinning your dependencies that you have for those functions, doing your typical code audits of what you are pulling in. Some of these serverless functions can also pull in data from a Pub/Sub, pull in data from GCS. So you're looking at where is that workload pulling data in to run? Because if someone has permission to write to that Pub/Sub, to write to that GCS, then they can impact not necessarily what that Cloud Function is doing, but the input to it.

So it sounds like it's integrity of data more than integrity of code. Like, to me, I agree with Matt, like the integrity of code, Seems like a far-fetched risk in this model, but integrity of data, such as corrupting something to write instead of reading, that still sounds kind of far-fetched to me. So we are back in the confidentiality or DoS risks, right? Like availability/DoS or data theft or exfil, right?

Yeah. And from the DoS perspective, where it gets tough is that it's not going to be a DoS like someone's hitting your WAN IP and knocking you offline. For a lot of these types of workloads, it will spin up and down as many instances as it needs to continue to be able to run. So a lot of the DoS, from what I've seen, is less about keeping your system from running as it is causing additional overhead and cost for you.

More of a denial of wallet than a denial of availability.

Right, right, like an economic DoS. Yeah, that's probably another example. I mean, it's not huge, but like an on-premise educated security engineer would not think in this direction right away. They would not go and think, oh, the DoS. No, it can't be a DoS. It's whole cloud. They can't DoS the cloud. But then you look at the bill and it's like your $700 became $700 million. Okay, that's an extreme example, but there can be a DoS along that parallel, right? Along that line.

Yeah. I love this term, DoS of wallet.

Yeah, DoS of wallet. And I will mention that there are lots of services in Google Cloud like billing alerts and budgets to help you, you know, be aware of these things.

Yeah, there's that. It's not unmanageable, but still a real challenge with some application designs from what I'm hearing. Yeah.

Oh, for sure. It's definitely a case of security engineers coming to cloud need to think with cloud instead of, you know, not thinking with portals.

So one thing to kind of shift gears a little bit to actual hows and what to monitor, like, I think in the modern stack we're describing, it sounds like the perimeter stuff is truly out of the picture? Like, is perimeter truly gone? Is this finally— we're in this perimeterless nirvana. We are doing application user identity. We are not doing perimeter. Is that true?

Are the firewalls resting in buckets?

Yeah. Are the firewalls just truly gone? Yes.

In this case, I think they're mostly gone if you keep yourself from having things like public cloud functions that anyone can hit. Then yes, we are in the cloud. No one can touch us except my S3 bucket is open to the public. Same type of situation where that config change can open you up to having a publicly exposed asset. Aside from that, you're absolutely right. It is breaking that entire paradigm of the perimeter because everything is now running as managed services within that SaaS environment, within that infrastructure.

Does that mean identity is the new perimeter?

You keep saying this.

Do you even know what it means?

I'm trying to figure out what it means.

I don't know if I'd say it's the new perimeter, but it is one of the most critical aspects to looking at how you secure your workloads because those identities are tied to so much access that then is honestly difficult to monitor. It is not easy to look through cloud audit logs to easily see in the hundreds of thousands of rows if this service account is acting goofy. Because that service account can pop up from dozens of IP addresses within your infrastructure, it can be making dozens of calls. So trying to see what is an outlier in a service account making a database call can be very tough because it can run on multiple workloads on different instances. instance types doing a whole lot of work, which is where, kind of going back to, that workload should have one identity that has a narrow scope, where that account is not doing a variety of functions.

You're saying that if somebody designs an application where there's an account that's used for many, many purposes, then it's already bad design. Like, you shouldn't really do it.

Absolutely. And this is not poking at GCP, but there is a default service account in GCP that can be used that has editor permission across the entire environment. It can edit, write, read anything. You can set up a Cloud Function and use that default service account as the actor in that workload. And I have seen it where that service account is accepting arbitrary commands in an HTTP request and executing them.

Yeah.

So what you have there is an ability to have arbitrary code execution across That's very 1990s.

I hope this would somehow magically be gone by now. It's almost like a shell command in a URL. It is.

Even if you don't look at like arbitrary code execution, if you look at that Cloud Function, if you have any way to give it bad data, it has such elevated permissions that it can do anything in your environment. So going back to the bad paradigm from the start, using service accounts like those default service accounts at all is a bad paradigm. whether you're looking at it from serverless or it running in GKE or any compute instance.

Okay.

Imagine you have an operation, you recently became sort of in charge of doing some of this monitoring. Like, what would be the steps you'd take? I think we talked about a lot of things, but we sort of never quite condensed them to 3 things to do if you are trying to do the detection response or monitoring for this environment. Like, what's your favorite tips?

Yeah, first one would be in GCP, you can look at each service account what APIs it's calling and which ones it hasn't called to strip out permissions that you don't need. So that's something that with a couple of clicks in Google's UI or within probably every other cloud vendor, you can go and look at what's not being used for that service account and start to descope some of those permissions.

Okay, but that's more about preparing the terrain for better monitoring. What about something that's actually involving detection and response? Like, this is kind of prep work to me, which is hugely useful, but it's not like the detection advice.

Looking at the audit logs, it'll get back to what I kind of touched on briefly, is looking at that service account trying to do things that you have not authorized it to do. So if you have a service account that now is trying to list every GCS bucket that you have in your environment and list the permissions for them, if it's not supposed to do that, that's weird. Someone's trying to enumerate all the buckets you have in your environment.

In MITRE terminology, that'd be a discovery tactic, huh? Yes.

Probably in the early stages of the attack, yes.

So we can look pretty far left for detection here. That's nice. What else can we look for, do you think?

What about somebody changing permissions? You would only see it from the code side. You'd never see it from people invoking functions side, right?

Yeah.

So I would say that's kind of to your point on like a far left thing. If you are architecting your infrastructure well, you should have checks for anyone manually modifying an IAM permission. So adding permissions to that account, And hopefully you have your infrastructure set up with IAM in code where it takes a PR, 2 people to do it, and someone to run that change. So same way as doing removal of permissions, you should be doing in-code permissions for IAM, validation of that, and watching for anyone manually doing it, 'cause you will have some SRE member that has to have editor permissions. You're never gonna get away from that world where someone has to be able to jump into the UI and fix things. So watching people like that doing things that they intend to be good, but that could open you up.

We, uh, I was at a startup a long time ago where we were extensive Google Cloud users. The YubiKey for the account that had org owner was kept in a safe. That's how we kept that locked down. I like the advice about managing permissions, you know, in some kind of checked-in repository with reviews on change and automated deploy against that. That's solid advice right there. I like that a lot.

And I think granular permissions was also somewhat basic from the, like, 1990s type of style security advice, but it's also much more implementable in the modern Yes. Because granular permissions on a typical mid-2000s Unix box is like a head-scratcher. Windows 2000—

Good luck.

A head-scratcher. Good luck. Yes, thank you. But here, when we say, hey, you can actually do granular permissions and then monitor for function use, that's actually actionable. It's much more actionable than what it was 10, 20 years ago, right?

Yeah. And it's not the pain of having to pass around a username and password through some backdoor system to land it on a machine. That workload just runs with the permission you gave it.

I'm so appreciative that we don't have to pass usernames and passwords around. I really ought to write a Valentine's Day card to whoever came up with that.

I'll send him the chocolates if you send him the card.

And at the same time, things like shared secrets, we still see an occasional use of shared secrets in the cloud. So how would you approach monitoring for secrets use in a modern stack? Would you expect to catch somebody, I don't know, calling a secret manager wrong? I don't know. What about the secrets use here?

Yeah, we definitely use Secret Manager, and the best practice that we have adopted is to, for that secret, only allow specific service accounts to access it. So the same way as looking for odd service account activity on your workloads, you are creating a specific service account that can access this one secret, and that's all that can access it. So you're looking for anyone else trying to reach out for that secret, and any permission denies on that. So you don't want to give someone generic secret reader permissions that they can access anything.

Right, because then we are back in the root password on the sticker world, just with modern tools, right? We're going to reimplement the problems of the '90s with the tools of 2020s, which I've seen people do, of course. It's hilarious unless you're living it.

Or doing the instant response to it.

Exactly, yeah. So let's see, we sort of hopefully build a bit of a structured view for people. Don't be afraid of the threat detection and monitoring challenges for a modern stack. There are things to do. There are certain broad patterns, identity, certain types of logging, shrinking permissions, being granular, watching for out-of-permission use. Any other principles or tricks that you want to share before we go to our final questions?

Yeah, I guess the last bit is really just that looking for effects. So if you think of looking at what the Cloud Function does, using the most basic example, if that Cloud Function takes a user ID and returns the user's information from a database? Well, you should be doing your standard looking for odd database calls, looking for malformed queries. So that is a very basic example, is looking at what does that cloud function do and touch in the backend of your infrastructure and looking for potential attempts to compromise those serverless architectures through what's going on in the backend.

That makes a lot of sense. So, that actually seems to answer half of our usual closing question, which is one pro tip for the audience to be better at this. The other question that we always ask so that we don't leave our listeners empty-handed is, do you have any reading that you could recommend for people to follow up to learn more about, you know, either thinking through these problems, responding to these problems, detecting these problems? What should people bone up on?

Yeah, actually, just before this, I was doing some Google searches to see how many different results you could get back if you just searched, in quotes, serverless security. And a couple months ago, I started to read through a bunch of them. There are a number of vendors that put things out. Palo Alto has a really good serverless security write-up. Check Point does as well. So those were 2 that were really helpful for me as I sat there and looked through it. You'll find, I would say, the same types of things in a lot of these where it's looking again at a lot of the stuff we touched on, looking at the identity, looking at the code review, looking at the data that those functions touch and securing those. So I think it kind of falls back to the same stuff that we talked about today when you're looking at those different articles.

Wait a second, code review? Isn't it way to the left of what we're discussing? Like, it sounds like we are talking about like monitoring what runs, but code review implies that you'd look at the code even if you're planning the operational activities. You wouldn't review code and then forget about it. Like, you would look at code once in a while during your detection planning.

So yes, you would do code reviews on the front end and should be doing code reviews as that code changes or as the architecture changes.

And it sounds like you should be assigning permissions in a code-based way where all permission updates get approved by somebody and reviewed by that person.

Yep, by that person and someone else. Yeah.

Don't forget that. By that person and somebody else. That's right.

Yeah.

Well, I read the SRE book as well, so I know that trick. I wouldn't fall for it. Not just that person. Okay, on that happy note, Thank you very much, Matt, and very much appreciate your time on the podcast.

Thank you very much. Appreciate it.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, Stitcher, and many other apps wherever you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. At this point, even I remember the URL, and you should too. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are also on Twitter, @anton_chuvakin and @teampeacock. Tweet at us, email us, argue with us. And if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.